Oct 30 20:58:41.393904 ip-10-0-139-90 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 20:58:41.393918 ip-10-0-139-90 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 20:58:41.393927 ip-10-0-139-90 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 20:58:41.394231 ip-10-0-139-90 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 20:58:51.642843 ip-10-0-139-90 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 20:58:51.642864 ip-10-0-139-90 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 384fa65c8f004c73a693f91d29a4faa8 --
Oct 30 21:00:45.292685 ip-10-0-139-90 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:00:45.723116 ip-10-0-139-90 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:00:45.723116 ip-10-0-139-90 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:00:45.723116 ip-10-0-139-90 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:00:45.723116 ip-10-0-139-90 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:00:45.731711 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731630    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:45.731711 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731711    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:45.731711 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731715    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731719    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731722    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731729    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731732    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731735    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731738    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731740    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731743    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731746    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731748    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731752    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731755    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731758    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731761    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731764    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731769    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731772    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731775    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:45.731808 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731777    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731780    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731783    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731785    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731788    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731790    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731793    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731796    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731798    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731803    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731806    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731809    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731812    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731815    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731818    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731821    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731823    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731826    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731829    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731832    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:45.732242 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731835    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731838    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731844    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731849    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731852    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731854    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731858    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731861    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731864    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731867    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731870    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731872    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731876    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731879    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731885    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731888    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731891    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731895    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731898    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731900    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:45.732726 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731903    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731906    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731908    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731911    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731913    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731916    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731980    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731985    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731988    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731991    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731993    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731996    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.731999    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732002    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732005    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732008    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732010    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732013    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732015    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732020    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:45.733191 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732024    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732027    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732030    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732033    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732035    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732038    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732041    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732044    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732047    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732050    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732053    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732056    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732061    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732063    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732066    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732068    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732074    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732077    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732081    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:45.733733 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732084    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:45.734171 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732086    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:45.734171 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732089    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:45.734171 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732091    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:45.734171 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732096    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:45.734171 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.732099    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:45.734907 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.734882    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:45.735169 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735154    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:45.735169 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735171    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735175    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735180    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735183    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735188    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735191    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735194    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735196    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735199    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735202    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735204    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735207    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735210    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735212    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735215    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735218    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735220    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735225    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735229    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735232    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:45.735280 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735235    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735237    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735240    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735243    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735246    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735249    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735251    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735255    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735257    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735260    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735262    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735265    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735267    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735269    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735273    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735276    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735278    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735283    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735286    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735289    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:45.735887 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735291    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735294    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735297    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735299    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735302    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735304    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735307    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735310    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735312    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735315    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735318    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735320    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735323    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735325    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735328    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735332    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735335    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735337    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735341    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:45.736357 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735344    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735349    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735351    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735354    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735357    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735361    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735366    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735370    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735374    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735377    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735381    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735384    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735404    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735408    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735413    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735418    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735421    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735423    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735426    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:45.736868 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735428    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735431    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735433    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735436    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735438    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735441    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735443    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735446    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735448    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735451    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735453    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735456    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735458    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735461    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735463    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735466    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735468    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735471    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735473    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735475    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:45.737318 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735479    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735482    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735485    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735487    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735491    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.735494    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736057    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736062    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736066    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736069    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736072    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736075    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736077    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736080    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736084    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736087    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736090    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736093    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736095    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:45.737819 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736098    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736100    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736103    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736105    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736108    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736110    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736112    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736115    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736118    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736120    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736125    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736128    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736130    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736133    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736136    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736138    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736140    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736143    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736145    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:45.738263 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736149    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736153    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736156    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736160    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736163    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736166    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736169    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736171    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736174    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736176    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736179    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736181    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736184    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736186    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736188    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736191    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736193    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736196    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736198    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736200    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:45.738718 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736203    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736206    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736208    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736213    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736215    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736218    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736221    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736224    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736226    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736229    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736231    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736234    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736236    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736239    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736241    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736244    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736247    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736250    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736252    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:45.739187 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736255    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736258    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736260    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736263    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736265    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736267    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736270    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736272    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736275    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736277    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736280    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736282    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736285    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736287    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736290    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736292    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736296    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736298    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736301    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:45.739658 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736304    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736306    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736309    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736311    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736314    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736316    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736319    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736321    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736324    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736327    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736329    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736332    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736334    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736338    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736340    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.736343    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737406    2582 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737416    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737423    2582 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737428    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737432    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:00:45.740133 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737435    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737440    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737444    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737447    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737450    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737454    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737457    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737460    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737463    2582 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737466    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737469    2582 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737472    2582 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737475    2582 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737480    2582 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737483    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737485    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737488    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737492    2582 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737496    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737499    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737502    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737505    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737508    2582 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737512    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:00:45.740638 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737515    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737518    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737520    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737524    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737527    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737530    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737533    2582 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737536    2582 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737539    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737543    2582 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737545    2582 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737548    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737552    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737554    2582 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737559    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737561    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737565    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737568    2582 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737572    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737575    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737578    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737580    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737583    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737586    2582 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737589    2582 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:00:45.741219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737599    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737602    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737605    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737608    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737611    2582 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737614    2582 flags.go:64] FLAG: --help="false"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737617    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737622    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737625    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737628    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737633    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737637    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737640    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737642    2582 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737645    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737648    2582 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737651    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737654    2582 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737657    2582 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737660    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737663    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737666    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737669    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:00:45.741869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737672    2582 flags.go:64] FLAG: --lock-file=""
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737674    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737677    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737681    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737687    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737690    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737692    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737695    2582 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737698    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737702    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737704    2582 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737707    2582 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737712    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737715    2582 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737719    2582 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737721    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737724    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737728    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737731    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737735    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737738    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737741    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737748    2582 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737751    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737754    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:00:45.742491 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737756    2582 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737759    2582 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737762    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737765    2582 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737768    2582 flags.go:64] FLAG: --port="10250"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737771    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737774    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-004282084a930176a"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737777    2582 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737780    2582 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737783    2582 flags.go:64] FLAG: --register-node="true"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737786    2582 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737790    2582 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737792    2582 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737795    2582 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737798    2582 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737802    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737804    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737807    2582 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737810    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737813    2582 flags.go:64] FLAG: --runonce="false"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737816    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737819    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737822    2582 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737825    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737828    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737832    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:00:45.743062 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737836    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737840    2582 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737842    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737845    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737848    2582 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737851    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737854    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737857    2582 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737859    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737864    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737867    2582 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737870    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737874    2582 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737878    2582 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737881    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737884    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737887    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737889    2582 flags.go:64] FLAG: --v="2"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737894    2582 flags.go:64] FLAG: --version="false"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737898    2582 flags.go:64] FLAG: --vmodule=""
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737902    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.737905    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738052    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738056    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738059    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:45.743740 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738061    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738064    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738067    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738069    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738072    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738074    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738077    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738083    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738086    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738088    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738091    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738093    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738096    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738099    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738102    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738106    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738109    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738112    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738114    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:45.744335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738117    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738121    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738124    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738126    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738128    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738131    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738134    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738136    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738138    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738141    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738144    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738146    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738149    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738152    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738154    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738157    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738159    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738162    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738164    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738167    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:45.744810 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738171    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738173    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738176    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738179    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738181    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738183    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738186    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738189    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738192    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738196    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738199    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738201    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738204    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738208    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738210    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738212    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738215    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738217    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738220    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:45.745275 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738223    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738225    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738228    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738230    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738233    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738235    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738238    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738240    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738243    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738245    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738248    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738251    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738253    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738257    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738259    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738265    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738268    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738270    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738273    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738276    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:45.745729 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738278    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738281    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738283    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738285    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738288    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738290    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738294    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738297    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738299    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738302    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738304    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738306    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738309    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738312    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738314    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738316    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738320    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738322    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738325    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:45.746209 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738327    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738330    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738332    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738335    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738337    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738340    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.738367    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738418    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738422    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738431    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738434    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738437    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738439    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738442    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738444    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:45.746673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738447    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738449    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738452    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738454    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738457    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738460    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738463    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738465    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738468    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738470    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738473    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738475    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738478    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738480    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738483    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738485    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738488    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738491    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738493    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738496    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:45.747018 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738498    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738501    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738503    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738506    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738508    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738510    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738513    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738517    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738519    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738522    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738524    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738527    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738529    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738532    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738534    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738537    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738539    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738543    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738545    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738548    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:45.747484 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738550    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738552    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738555    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738558    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738560    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738563    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738565    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738568    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738571    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738573    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738576    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738578    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738581    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738583    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738587    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738591    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738594    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738598    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738601    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:45.747952 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738603    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738606    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738609    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738611    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738614    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738616    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738619    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738621    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738623    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738626    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738629    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738632    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738634    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738637    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738639    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738642    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738644    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738647    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738649    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738652    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:45.748413 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738654    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738657    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738659    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738662    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738664    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738666    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738669    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738671    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738675    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738679    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738682    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738685    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738688    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738690    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738693    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738696    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738698    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738701    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.738704    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:45.748922 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.746526    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.746541    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.746558    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.746562    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746625    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746631    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746634    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746637    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746640    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746642    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746645    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746648    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746650    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746653    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746655    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746658    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746660    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746663    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746666    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746668    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:45.749371 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746671    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746673    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746676    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746679    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746682    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746685    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746687    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746691    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746694    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746696    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746698    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746701    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746703    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746706    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746709    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746711    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746714    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746716    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746719    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:45.749894 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746721    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746724    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746728    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746731    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746735    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746738    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746740    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746743    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746745    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746747    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746750    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746753    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746755    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746758    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746760    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746762    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746765    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746768    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746770    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:45.750335 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746772    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746775    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746778    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746781    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746784    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746786    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746789    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746792    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746794    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746797    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746799    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746802    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746804    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746807    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746810    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746814    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746817    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746820    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746823    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746825    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:45.750787 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746828    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746831    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746833    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746836    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746838    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746841    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746843    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746845    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746848    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746850    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746853    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746856    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746858    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746861    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746864    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746867    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746870    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746872    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746875    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:45.751277 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746878    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746880    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746883    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746885    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746887    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746890    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746892    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746895    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746897    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746899    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746902    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746904    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.746907    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.746934    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747107    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747113    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:45.751746 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747116    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747119    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747122    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747124    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747127    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747129    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747132    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747134    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747137    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747139    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747142    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747144    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747147    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747150    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747153    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747155    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747159    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747161    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747164    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:45.752108 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747167    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747169    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747171    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747174    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747177    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747180    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747182    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747185    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747188    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747190    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747193    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747195    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747198    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747200    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747202    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747205    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747208    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747210    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747213    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:45.752572 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747215    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747217    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747220    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747222    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747225    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747228    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747230    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747233    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747236    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747238    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747241    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747243    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747246    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747248    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747250    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747253    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747257    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747261    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747264    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:45.753002 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747267    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747269    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747273    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747276    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747278    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747281    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747283    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747285    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747288    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747291    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747294    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747296    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747299    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747301    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747304    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747308    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747312    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747314    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747317    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747320    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:45.753444 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747322    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747325    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747328    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747331    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747333    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747336    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747338    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747341    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747343    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747346    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747348    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747351    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747353    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747356    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747358    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747361    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747363    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747365    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747368    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747370    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:45.753928 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747372    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747375    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747377    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747380    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747382    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747384    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747400    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.747436    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747482    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747486    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747490    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747493    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747496    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747499    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747501    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747511    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:00:45.754402 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747514    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747517    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747520    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747522    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747525    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747527    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747530    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747533    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747541    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747546    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747549    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747552    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747554    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747557    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747559    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747562    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747565    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747568    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747570    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:00:45.754771 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747573    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747575    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747578    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747580    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747583    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747585    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747588    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747591    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747593    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747596    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747598    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747600    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747603    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747611    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747614    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747617    2582 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747619    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747622    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747624    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:00:45.755218 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747627    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747630    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747634    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747637    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747639    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747642    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747644    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747647    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747649    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747652    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747655    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747657    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747660    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747662    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747664    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747667    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747669    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747671    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747674    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:00:45.755678 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747679    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747682    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747685    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747688    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747691    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747694    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747696    2582 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747699    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747707    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747710    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747713    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747715    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747717    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747720    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747722    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747724    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747731    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747734    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747736    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747738    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:00:45.756142 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747741    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747744    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747746    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747749    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747751    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747753    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747756    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747758    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747761    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747763    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747766    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747771    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747774    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747777    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747779    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747782    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747784    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747787    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747789    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747792    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:00:45.756673 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:45.747794    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:00:45.757137 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.747854    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:00:45.757137 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.750372    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:00:45.757137 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.751471    2582 server.go:1014] "Starting client certificate rotation"
Oct 30 21:00:45.757137 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.751577    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:00:45.757137 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.751618    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:00:45.757137 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.753831    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:00:45.757137 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.755123    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:00:45.764531 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.764512    2582 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:00:45.769095 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.769078    2582 log.go:25] "Validated CRI v1 image API"
Oct 30 21:00:45.769982 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.769946    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:00:45.777467 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.777445    2582 fs.go:135] Filesystem UUIDs: map[1381bd88-137e-4352-80bd-e5458225a786:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8662b3f6-4470-4a8e-8c4e-b3f0b1fa4932:/dev/nvme0n1p3]
Oct 30 21:00:45.777534 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.777467    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:00:45.783232 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.783120    2582 manager.go:217] Machine: {Timestamp:2026-10-30 21:00:45.781533013 +0000 UTC m=+0.386477145 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096227 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2815ce5eae4c005e02a3f42cd38b47 SystemUUID:ec2815ce-5eae-4c00-5e02-a3f42cd38b47 BootID:384fa65c-8f00-4c73-a693-f91d29a4faa8 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e3:89:72:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e3:89:72:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:c9:3c:a2:b5:99 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:00:45.783868 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.783858    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:00:45.783993 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.783982    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:00:45.784268 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.784243    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:00:45.786806 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.786779    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:00:45.786939 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.786807    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-90.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:00:45.786982 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.786942    2582 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:00:45.786982 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.786950    2582 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:00:45.786982 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.786967    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:00:45.787056 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.786982    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:00:45.788221 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.788211    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:00:45.792808 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.792797    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:00:45.792931 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.792921    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:00:45.792966 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.792958    2582 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:00:45.792998 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.792970    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:00:45.792998 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.792984    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:00:45.793131 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.792999    2582 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:00:45.793131 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.793008    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:00:45.793344 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.793332    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:00:45.795792 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.795774    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:00:45.802246 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.802230    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:00:45.803295 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.803281    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:00:45.803385 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.803321    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:00:45.803592 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.803577    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:00:45.803668 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.803599    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:00:45.803668 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.803609    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:00:45.803668 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.803618    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:00:45.803668 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.803627    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:00:45.803668 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.803636    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:00:45.804294 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.804282    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:00:45.804349 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.804298    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:00:45.804349 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.804333    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:00:45.804349 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.804344    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:00:45.804498 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.804357    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:00:45.804498 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.804373    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:00:45.805515 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.805503    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:00:45.805565 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.805519    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:00:45.809560 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.809544    2582 server.go:1263] "Started kubelet"
Oct 30 21:00:45.809626 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.809586    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:00:45.809713 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.809673    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:00:45.809771 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.809728    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:00:45.811318 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.811295    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:00:45.811939 ip-10-0-139-90 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:00:45.814676 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.814655    2582 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:00:45.817301 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.817264    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:00:45.820174 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.820155    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:00:45.820792 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.820772    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:00:45.820886 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.820774    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:00:45.820886 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.820881    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-90.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:00:45.821205 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.821183    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:45.821372 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.821347    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-90.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 21:00:45.821607 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.821592    2582 factory.go:55] Registering systemd factory
Oct 30 21:00:45.821607 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.821608    2582 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:00:45.821726 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.821653    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:00:45.821834 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.821818    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:00:45.821869 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.821840    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:00:45.821916 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.821820    2582 factory.go:153] Registering CRI-O factory
Oct 30 21:00:45.821944 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.821924    2582 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:00:45.821979 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.821969    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:00:45.822014 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.821990    2582 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:00:45.822014 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.821994    2582 factory.go:103] Registering Raw factory
Oct 30 21:00:45.822014 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.822012    2582 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:00:45.822173 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.821997    2582 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:00:45.822505 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.821330    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-90.us-east-2.compute.internal.18e36a6202c5c677  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-90.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-90.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:00:45.809444471 +0000 UTC m=+0.414388600,LastTimestamp:2026-10-30 21:00:45.809444471 +0000 UTC m=+0.414388600,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-90.us-east-2.compute.internal,}"
Oct 30 21:00:45.822969 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.822949    2582 manager.go:319] Starting recovery of all containers
Oct 30 21:00:45.830133 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.830098    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-90.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:00:45.833306 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.833273    2582 manager.go:324] Recovery completed
Oct 30 21:00:45.835028 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.834995    2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 21:00:45.835155 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.835136    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 21:00:45.837716 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.837705    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:45.839270 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.839252    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:45.839334 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.839287    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:45.839334 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.839298    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:45.839842 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.839824    2582 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:00:45.839842 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.839840    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:00:45.839947 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.839861    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:45.839947 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.839859    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:00:45.840548 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.840534    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:45.840609 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.840559    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:45.840609 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.840570    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:45.841933 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.841872    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-90.us-east-2.compute.internal.18e36a62048cda1b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-90.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-90.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-90.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:00:45.839268379 +0000 UTC m=+0.444212509,LastTimestamp:2026-10-30 21:00:45.839268379 +0000 UTC m=+0.444212509,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-90.us-east-2.compute.internal,}"
Oct 30 21:00:45.842077 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.842060    2582 policy_none.go:50] "Start"
Oct 30 21:00:45.842145 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.842081    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:00:45.842145 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.842092    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:00:45.844172 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.844159    2582 policy_none.go:44] "Start"
Oct 30 21:00:45.845368 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.845353    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zmt5h"
Oct 30 21:00:45.851315 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.851248    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-90.us-east-2.compute.internal.18e36a62048d36c2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-90.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-90.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-90.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:00:45.839292098 +0000 UTC m=+0.444236228,LastTimestamp:2026-10-30 21:00:45.839292098 +0000 UTC m=+0.444236228,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-90.us-east-2.compute.internal,}"
Oct 30 21:00:45.857964 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.857888    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-90.us-east-2.compute.internal.18e36a62048d5de0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-90.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-90.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-90.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:00:45.839302112 +0000 UTC m=+0.444246244,LastTimestamp:2026-10-30 21:00:45.839302112 +0000 UTC m=+0.444246244,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-90.us-east-2.compute.internal,}"
Oct 30 21:00:45.858076 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.858058    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zmt5h"
Oct 30 21:00:45.865370 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.865199    2582 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-90.us-east-2.compute.internal.18e36a62048cda1b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-90.us-east-2.compute.internal.18e36a62048cda1b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-90.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-90.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-90.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:00:45.839268379 +0000 UTC m=+0.444212509,LastTimestamp:2026-10-30 21:00:45.840547205 +0000 UTC m=+0.445491342,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-90.us-east-2.compute.internal,}"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.879011    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.881654    2582 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.881680    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.881692    2582 server.go:87] "Starting device plugin registration server"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.881931    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.881940    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.882049    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.882134    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.882143    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.882698    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.882745    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.892797    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.894062    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.894689    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.894718    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.894754    2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:00:45.901751 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.898726    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:45.982987 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.982887    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:45.985128 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.985108    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:45.985251 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.985138    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:45.985251 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.985148    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:45.985251 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.985175    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:45.995876 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.995855    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"]
Oct 30 21:00:45.995982 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.995923    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:45.996902 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.996887    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:45.996983 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.996916    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:45.996983 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.996925    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:45.997121 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.997102    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:45.997227 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:45.997126    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-90.us-east-2.compute.internal\": node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:45.999194 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.999182    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:45.999343 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.999326    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:45.999384 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.999362    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:45.999971 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.999956    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:46.000064 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.999978    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:46.000064 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:45.999992    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:46.000064 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.000045    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:46.000064 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.000062    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:46.000186 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.000072    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:46.002167 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.002154    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.002237 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.002177    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:46.002885 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.002872    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:46.002953 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.002900    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:46.002953 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.002912    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:46.022587 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:46.022560    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:46.023163 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.023145    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb77e2dfdbc4a16f3bcede022ac475a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"bb77e2dfdbc4a16f3bcede022ac475a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.023211 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.023172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.023211 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.023188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.034113 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:46.034093    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" node="ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.036063 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:46.036045    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found" node="ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.123141 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:46.123115    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:46.124231 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.124207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb77e2dfdbc4a16f3bcede022ac475a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"bb77e2dfdbc4a16f3bcede022ac475a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.124361 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.124247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.124361 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.124266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.124361 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.124312    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.124361 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.124316    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.124361 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.124317    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb77e2dfdbc4a16f3bcede022ac475a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"bb77e2dfdbc4a16f3bcede022ac475a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.224022 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:46.223971    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:46.324823 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:46.324753    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:46.339012 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.338993    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.341102 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.341080    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:46.425895 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:46.425865    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:46.526278 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:46.526243    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:46.626705 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:46.626635    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:46.727043 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:46.727014    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:46.751209 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.751183    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:00:46.751353 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.751335    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:00:46.827546 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:46.827514    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:46.864673 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.864640    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:55:45 +0000 UTC" deadline="2028-07-05 17:44:11.490064908 +0000 UTC"
Oct 30 21:00:46.864673 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.864671    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14732h43m24.625397025s"
Oct 30 21:00:46.900943 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.900884    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:46.927981 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:46.927957    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:46.934419 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.934401    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:46.952479 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.952460    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:00:46.954224 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.954209    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:00:46.954310 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.954239    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:00:46.954310 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:46.954249    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:00:47.028758 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:47.028726    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:47.105029 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:47.104992    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fda1c39c64a6789fa2412aae915cee.slice/crio-d5d62d81a997ab8ff4fa9fabcd11777b9f2d3f56bdb052936172689e0b460c31 WatchSource:0}: Error finding container d5d62d81a997ab8ff4fa9fabcd11777b9f2d3f56bdb052936172689e0b460c31: Status 404 returned error can't find the container with id d5d62d81a997ab8ff4fa9fabcd11777b9f2d3f56bdb052936172689e0b460c31
Oct 30 21:00:47.105196 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:47.105176    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb77e2dfdbc4a16f3bcede022ac475a7.slice/crio-bb80dbb42c8818d28f29cbf26f14c62249c49980f775ae61757668f9f8e19c5b WatchSource:0}: Error finding container bb80dbb42c8818d28f29cbf26f14c62249c49980f775ae61757668f9f8e19c5b: Status 404 returned error can't find the container with id bb80dbb42c8818d28f29cbf26f14c62249c49980f775ae61757668f9f8e19c5b
Oct 30 21:00:47.111555 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.111531    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:00:47.129704 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:47.129687    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:00:47.139405 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.139372    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:47.221546 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.221469    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:47.237406 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.237375    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:00:47.238503 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.238490    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:00:47.247314 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.247300    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:00:47.793608 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.793575    2582 apiserver.go:52] "Watching apiserver"
Oct 30 21:00:47.800253 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.800228    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:00:47.800650 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.800628    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-lwgsc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal","openshift-network-operator/iptables-alerter-zbdld","openshift-ovn-kubernetes/ovnkube-node-zb8qk","kube-system/global-pull-secret-syncer-qth9k","openshift-cluster-node-tuning-operator/tuned-2z8lm","openshift-dns/node-resolver-q7r7r","openshift-multus/multus-n8kk5","openshift-multus/network-metrics-daemon-xppxb","kube-system/konnectivity-agent-t48nt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp","openshift-network-diagnostics/network-check-target-6slln","kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal","openshift-image-registry/node-ca-rzn4z"]
Oct 30 21:00:47.803381 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.803357    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:00:47.805716 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.805697    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.807294 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.807265    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:00:47.807422 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.807347    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mwc48\""
Oct 30 21:00:47.807490 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.807441    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:00:47.808667 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.808644    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:00:47.809355 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.809337    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:00:47.809473 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.809463    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:00:47.809539 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.809520    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:00:47.809673 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.809621    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qfb4q\""
Oct 30 21:00:47.811129 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.811107    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.811704 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.811686    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qbtrh\""
Oct 30 21:00:47.811799 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.811747    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:00:47.811919 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.811870    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:00:47.812019 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.812000    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:00:47.813550 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.813529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:47.813643 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:47.813603    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:00:47.814656 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.814634    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x8hl8\""
Oct 30 21:00:47.814791 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.814776    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:00:47.814922 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.814909    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:00:47.814977 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.814960    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:00:47.815473 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.815161    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:00:47.815473 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.815232    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:00:47.815473 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.815356    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:00:47.816923 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.816906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.819451 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.819431    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:47.819547 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:47.819505    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:00:47.821311 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.821291    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:00:47.822512 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.822474    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:00:47.824534 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.824513    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.827180 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.827168    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:00:47.829978 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.829935    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:47.832940 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.832920    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:47.833036 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:47.832989    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:00:47.833284 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833265    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-registration-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.833346 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833299    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-run-netns\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.833346 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-node-log\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.833509 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jl6z\" (UniqueName: \"kubernetes.io/projected/829cdf21-2065-4e39-902d-7b857a2d74bd-kube-api-access-8jl6z\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:00:47.833509 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4gw\" (UniqueName: \"kubernetes.io/projected/3a08d3f6-f560-4507-9596-9b548981063c-kube-api-access-tn4gw\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.833509 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833432    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.833509 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-sys\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.833509 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-host\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.833815 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833524    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-systemd\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.833815 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-cni-bin\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.833815 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-cni-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.833815 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833610    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-etc-kubernetes\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.833815 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-device-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.833815 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-var-lib-kubelet\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.833815 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l52q4\" (UniqueName: \"kubernetes.io/projected/b459c3a5-a5d9-4776-9788-a7d7b943fa53-kube-api-access-l52q4\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:00:47.833815 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.833815 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-etc-selinux\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.833815 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833766    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysctl-d\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.833815 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833789    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-lib-modules\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.833815 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-ovnkube-config\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-ovnkube-script-lib\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833858    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-cni-multus\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e6a92-3eef-423d-859e-405adb01c3a1-tmp-dir\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6n67\" (UniqueName: \"kubernetes.io/projected/98891442-5d51-4e57-a74a-f0fc22de1113-kube-api-access-n6n67\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-kubelet-config\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5jp8\" (UniqueName: \"kubernetes.io/projected/d80303a1-1861-4958-a037-891078b48456-kube-api-access-w5jp8\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.833998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-etc-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/829cdf21-2065-4e39-902d-7b857a2d74bd-serviceca\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834072    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-modprobe-d\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysctl-conf\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjhl2\" (UniqueName: \"kubernetes.io/projected/31769be6-0719-4533-997c-e8b458c54d79-kube-api-access-kjhl2\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834163    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-netns\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834190    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a08d3f6-f560-4507-9596-9b548981063c-multus-daemon-config\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-kubernetes\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.834387 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834238    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-tuned\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b459c3a5-a5d9-4776-9788-a7d7b943fa53-host-slash\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834310    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-kubelet\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-systemd\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834357    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-run\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.834412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-ovn\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835087    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-cni-bin\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835241    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31769be6-0719-4533-997c-e8b458c54d79-ovn-node-metrics-cert\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsq84\" (UniqueName: \"kubernetes.io/projected/f356dcb1-1e78-4428-bc1f-43ede09e47e3-kube-api-access-vsq84\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-cni-netd\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835316    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-socket-dir-parent\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835340    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-k8s-cni-cncf-io\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-hostroot\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835386    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-dbus\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:47.836454 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f356dcb1-1e78-4428-bc1f-43ede09e47e3-tmp\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-kubelet\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/829cdf21-2065-4e39-902d-7b857a2d74bd-host\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-cnibin\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-multus-certs\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835569    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk72h\" (UniqueName: \"kubernetes.io/projected/082e6a92-3eef-423d-859e-405adb01c3a1-kube-api-access-wk72h\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-sys-fs\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-var-lib-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.835972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-run-ovn-kubernetes\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.836008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a08d3f6-f560-4507-9596-9b548981063c-cni-binary-copy\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.836180    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-system-cni-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.836209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-socket-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.836242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysconfig\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.836456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/082e6a92-3eef-423d-859e-405adb01c3a1-hosts-file\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.836494    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b459c3a5-a5d9-4776-9788-a7d7b943fa53-iptables-alerter-script\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.836526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-slash\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.836708    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-log-socket\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.837519 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.836743    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-env-overrides\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.838368 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.836848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-systemd-units\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.838368 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.836883    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-conf-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.855782 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.855736    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bxdsg"
Oct 30 21:00:47.863557 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.863536    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bxdsg"
Oct 30 21:00:47.901588 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.901522    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"bb77e2dfdbc4a16f3bcede022ac475a7","Type":"ContainerStarted","Data":"bb80dbb42c8818d28f29cbf26f14c62249c49980f775ae61757668f9f8e19c5b"}
Oct 30 21:00:47.905551 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.905525    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"29fda1c39c64a6789fa2412aae915cee","Type":"ContainerStarted","Data":"d5d62d81a997ab8ff4fa9fabcd11777b9f2d3f56bdb052936172689e0b460c31"}
Oct 30 21:00:47.912330 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912306    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:00:47.912448 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912331    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:00:47.912448 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912399    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:00:47.912558 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912512    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:00:47.912629 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912606    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:00:47.912694 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912679    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-99mk5\""
Oct 30 21:00:47.912754 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912734    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:00:47.912806 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912784    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v7745\""
Oct 30 21:00:47.912852 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912835    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:00:47.912852 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912843    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:00:47.912938 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912606    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wx578\""
Oct 30 21:00:47.912938 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912737    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:00:47.913046 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912950    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:00:47.913046 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.912971    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hspc2\""
Oct 30 21:00:47.913349 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.913331    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:00:47.913418 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.913383    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:00:47.913587 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.913465    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g68zh\""
Oct 30 21:00:47.913587 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.913526    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:00:47.922986 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.922956    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:00:47.937237 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-conf-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.937332 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-system-cni-dir\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:47.937332 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-registration-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.937466 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937344    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-run-netns\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.937466 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-node-log\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.937466 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jl6z\" (UniqueName: \"kubernetes.io/projected/829cdf21-2065-4e39-902d-7b857a2d74bd-kube-api-access-8jl6z\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:00:47.937466 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4gw\" (UniqueName: \"kubernetes.io/projected/3a08d3f6-f560-4507-9596-9b548981063c-kube-api-access-tn4gw\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.937669 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937471    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.937669 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-sys\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.937669 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937520    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-host\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.937669 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-systemd\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.937669 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937589    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e6f376c-f439-45e5-9049-d7d097391b83-konnectivity-ca\") pod \"konnectivity-agent-t48nt\" (UID: \"4e6f376c-f439-45e5-9049-d7d097391b83\") " pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:00:47.937669 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-cni-bin\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.937669 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-cni-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.937669 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-etc-kubernetes\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.937669 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-device-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-var-lib-kubelet\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l52q4\" (UniqueName: \"kubernetes.io/projected/b459c3a5-a5d9-4776-9788-a7d7b943fa53-kube-api-access-l52q4\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-etc-selinux\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysctl-d\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-lib-modules\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-ovnkube-config\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937844    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-ovnkube-script-lib\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-cni-multus\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e6a92-3eef-423d-859e-405adb01c3a1-tmp-dir\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6n67\" (UniqueName: \"kubernetes.io/projected/98891442-5d51-4e57-a74a-f0fc22de1113-kube-api-access-n6n67\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-kubelet-config\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5jp8\" (UniqueName: \"kubernetes.io/projected/d80303a1-1861-4958-a037-891078b48456-kube-api-access-w5jp8\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.937984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-etc-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/829cdf21-2065-4e39-902d-7b857a2d74bd-serviceca\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:00:47.938096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl567\" (UniqueName: \"kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567\") pod \"network-check-target-6slln\" (UID: \"684225a2-9b4e-4c1f-a784-85da7509bbf1\") " pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-modprobe-d\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysctl-conf\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938093    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhl2\" (UniqueName: \"kubernetes.io/projected/31769be6-0719-4533-997c-e8b458c54d79-kube-api-access-kjhl2\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-netns\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a08d3f6-f560-4507-9596-9b548981063c-multus-daemon-config\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938146    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-kubernetes\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938162    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-tuned\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b459c3a5-a5d9-4776-9788-a7d7b943fa53-host-slash\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938210    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-kubelet\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-systemd\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-run\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-ovn\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.938911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-cni-bin\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31769be6-0719-4533-997c-e8b458c54d79-ovn-node-metrics-cert\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938359    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e6f376c-f439-45e5-9049-d7d097391b83-agent-certs\") pod \"konnectivity-agent-t48nt\" (UID: \"4e6f376c-f439-45e5-9049-d7d097391b83\") " pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsq84\" (UniqueName: \"kubernetes.io/projected/f356dcb1-1e78-4428-bc1f-43ede09e47e3-kube-api-access-vsq84\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-cni-netd\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-socket-dir-parent\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-k8s-cni-cncf-io\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-hostroot\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938467    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj6f4\" (UniqueName: \"kubernetes.io/projected/86d1bc79-bd2e-422f-b8aa-6de92eabb634-kube-api-access-bj6f4\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-dbus\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f356dcb1-1e78-4428-bc1f-43ede09e47e3-tmp\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938518    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-kubelet\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/829cdf21-2065-4e39-902d-7b857a2d74bd-host\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-cnibin\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-multus-certs\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk72h\" (UniqueName: \"kubernetes.io/projected/082e6a92-3eef-423d-859e-405adb01c3a1-kube-api-access-wk72h\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938615    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:47.939737 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-sys-fs\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-var-lib-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-run-ovn-kubernetes\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a08d3f6-f560-4507-9596-9b548981063c-cni-binary-copy\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938708    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cnibin\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938723    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-system-cni-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-socket-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysconfig\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/082e6a92-3eef-423d-859e-405adb01c3a1-hosts-file\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b459c3a5-a5d9-4776-9788-a7d7b943fa53-iptables-alerter-script\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-slash\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-log-socket\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-env-overrides\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-systemd-units\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938945    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-systemd-units\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.938977    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-conf-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.940615 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.939029    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-registration-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.939050    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-run-netns\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.939072    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-node-log\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.939415    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.939470    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-sys\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.939506    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-host\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.939536    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-systemd\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.939593    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-cni-bin\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.939801    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-cni-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.939839    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-etc-kubernetes\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.939882    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-device-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.939924    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-var-lib-kubelet\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.940130    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.940241    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-etc-selinux\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.940313    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysctl-d\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.940372    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-lib-modules\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.940858    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-ovnkube-config\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.941408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.941293    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-cni-multus\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.942224 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.941497    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysconfig\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.942224 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.941542    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-var-lib-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.942224 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.941583    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-run-ovn-kubernetes\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.942224 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.941719    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-system-cni-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.942224 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.941878    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-cnibin\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.942224 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.941918    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-slash\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.942224 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.941928    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-kubelet\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.942224 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.941957    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/829cdf21-2065-4e39-902d-7b857a2d74bd-host\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:00:47.942224 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.942010    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/082e6a92-3eef-423d-859e-405adb01c3a1-hosts-file\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:00:47.942224 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.941425    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-sys-fs\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.942224 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.942044    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-multus-certs\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.942764 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.942312    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-kubernetes\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.942764 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.942597    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-kubelet-config\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:47.942865 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.942847    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-etc-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.943331 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.943303    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-ovnkube-script-lib\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.943459 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.943395    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-log-socket\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.943526 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.943476    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-systemd\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.943858 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.943838    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-env-overrides\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.944013 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.943997    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysctl-conf\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.944477 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:47.944109    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:47.944477 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:47.944180    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret podName:b941fd2b-fbbd-4343-81e1-9a0cdef17a10 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:48.444159703 +0000 UTC m=+3.049103841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret") pod "global-pull-secret-syncer-qth9k" (UID: "b941fd2b-fbbd-4343-81e1-9a0cdef17a10") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:47.944477 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.944405    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-modprobe-d\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.944700 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.944481    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-ovn\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.944700 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.944534    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-run\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.944700 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.944574    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.945075 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.945052    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e6a92-3eef-423d-859e-405adb01c3a1-tmp-dir\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:00:47.945269 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.945170    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-cni-bin\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.945269 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.945223    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b459c3a5-a5d9-4776-9788-a7d7b943fa53-host-slash\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:00:47.945392 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:47.945295    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:47.945392 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.945346    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a08d3f6-f560-4507-9596-9b548981063c-cni-binary-copy\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.945392 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:47.945365    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs podName:d80303a1-1861-4958-a037-891078b48456 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:48.445339423 +0000 UTC m=+3.050283557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs") pod "network-metrics-daemon-xppxb" (UID: "d80303a1-1861-4958-a037-891078b48456") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:47.945580 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.945423    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-cni-netd\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.945580 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.945423    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-netns\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.945580 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.945493    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-k8s-cni-cncf-io\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.945580 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.945516    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-socket-dir-parent\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.945580 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.945564    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-kubelet\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.945888 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.945778    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-hostroot\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.945888 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.945849    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-dbus\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:47.946094 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.946076    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a08d3f6-f560-4507-9596-9b548981063c-multus-daemon-config\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.946279 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.946217    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-socket-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:47.947252 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.946481    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:00:47.947252 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.946639    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/829cdf21-2065-4e39-902d-7b857a2d74bd-serviceca\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:00:47.947252 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.946664    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b459c3a5-a5d9-4776-9788-a7d7b943fa53-iptables-alerter-script\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:00:47.952105 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.952078    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f356dcb1-1e78-4428-bc1f-43ede09e47e3-tmp\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.952105 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.952095    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-tuned\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.952274 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.952163    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31769be6-0719-4533-997c-e8b458c54d79-ovn-node-metrics-cert\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.952332 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.952270    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jl6z\" (UniqueName: \"kubernetes.io/projected/829cdf21-2065-4e39-902d-7b857a2d74bd-kube-api-access-8jl6z\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:00:47.954738 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.954694    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsq84\" (UniqueName: \"kubernetes.io/projected/f356dcb1-1e78-4428-bc1f-43ede09e47e3-kube-api-access-vsq84\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:47.955860 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.955214    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l52q4\" (UniqueName: \"kubernetes.io/projected/b459c3a5-a5d9-4776-9788-a7d7b943fa53-kube-api-access-l52q4\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:00:47.955860 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.955603    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk72h\" (UniqueName: \"kubernetes.io/projected/082e6a92-3eef-423d-859e-405adb01c3a1-kube-api-access-wk72h\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:00:47.961026 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.961004    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5jp8\" (UniqueName: \"kubernetes.io/projected/d80303a1-1861-4958-a037-891078b48456-kube-api-access-w5jp8\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:47.961609 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.961558    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4gw\" (UniqueName: \"kubernetes.io/projected/3a08d3f6-f560-4507-9596-9b548981063c-kube-api-access-tn4gw\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:47.966697 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.966672    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjhl2\" (UniqueName: \"kubernetes.io/projected/31769be6-0719-4533-997c-e8b458c54d79-kube-api-access-kjhl2\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:47.968074 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:47.968053    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6n67\" (UniqueName: \"kubernetes.io/projected/98891442-5d51-4e57-a74a-f0fc22de1113-kube-api-access-n6n67\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:48.039759 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.039719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e6f376c-f439-45e5-9049-d7d097391b83-konnectivity-ca\") pod \"konnectivity-agent-t48nt\" (UID: \"4e6f376c-f439-45e5-9049-d7d097391b83\") " pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:00:48.039929 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.039769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.039929 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.039804    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl567\" (UniqueName: \"kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567\") pod \"network-check-target-6slln\" (UID: \"684225a2-9b4e-4c1f-a784-85da7509bbf1\") " pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:48.039929 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.039854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.039929 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.039879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e6f376c-f439-45e5-9049-d7d097391b83-agent-certs\") pod \"konnectivity-agent-t48nt\" (UID: \"4e6f376c-f439-45e5-9049-d7d097391b83\") " pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:00:48.039929 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.039904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj6f4\" (UniqueName: \"kubernetes.io/projected/86d1bc79-bd2e-422f-b8aa-6de92eabb634-kube-api-access-bj6f4\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.040188 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.039932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.040188 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.039958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cnibin\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.040188 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.039988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.040188 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.040013    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-system-cni-dir\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.040188 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.040087    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-system-cni-dir\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.040478 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.040452    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e6f376c-f439-45e5-9049-d7d097391b83-konnectivity-ca\") pod \"konnectivity-agent-t48nt\" (UID: \"4e6f376c-f439-45e5-9049-d7d097391b83\") " pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:00:48.040566 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.040540    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cnibin\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.040646 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.040625    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.040787 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.040759    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.042857 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.042835    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.042986 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.042967    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.043579 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.043562    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e6f376c-f439-45e5-9049-d7d097391b83-agent-certs\") pod \"konnectivity-agent-t48nt\" (UID: \"4e6f376c-f439-45e5-9049-d7d097391b83\") " pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:00:48.046099 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:48.046053    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:00:48.046099 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:48.046075    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:00:48.046099 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:48.046087    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fl567 for pod openshift-network-diagnostics/network-check-target-6slln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:48.046299 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:48.046149    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567 podName:684225a2-9b4e-4c1f-a784-85da7509bbf1 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:48.546130773 +0000 UTC m=+3.151074908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fl567" (UniqueName: "kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567") pod "network-check-target-6slln" (UID: "684225a2-9b4e-4c1f-a784-85da7509bbf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:48.049162 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.049134    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj6f4\" (UniqueName: \"kubernetes.io/projected/86d1bc79-bd2e-422f-b8aa-6de92eabb634-kube-api-access-bj6f4\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.125162 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.124887    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:00:48.137293 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:48.137260    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082e6a92_3eef_423d_859e_405adb01c3a1.slice/crio-44cc001c54d126f5b3bad814f43e7bef6849925cecb8176809e742a8ae8ab773 WatchSource:0}: Error finding container 44cc001c54d126f5b3bad814f43e7bef6849925cecb8176809e742a8ae8ab773: Status 404 returned error can't find the container with id 44cc001c54d126f5b3bad814f43e7bef6849925cecb8176809e742a8ae8ab773
Oct 30 21:00:48.139893 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.139868    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:00:48.151235 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:48.151205    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98891442_5d51_4e57_a74a_f0fc22de1113.slice/crio-1933d1d2a61246b566d9781890abf1d4841a4f89cd7aa43ab6763d5b4f0005ac WatchSource:0}: Error finding container 1933d1d2a61246b566d9781890abf1d4841a4f89cd7aa43ab6763d5b4f0005ac: Status 404 returned error can't find the container with id 1933d1d2a61246b566d9781890abf1d4841a4f89cd7aa43ab6763d5b4f0005ac
Oct 30 21:00:48.158278 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.158256    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:00:48.169252 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:48.169224    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb459c3a5_a5d9_4776_9788_a7d7b943fa53.slice/crio-efb857e7821bb8b237f793852c65f05e1232206434296b808d46c55e506bb33f WatchSource:0}: Error finding container efb857e7821bb8b237f793852c65f05e1232206434296b808d46c55e506bb33f: Status 404 returned error can't find the container with id efb857e7821bb8b237f793852c65f05e1232206434296b808d46c55e506bb33f
Oct 30 21:00:48.178286 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.178264    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:00:48.183549 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.183524    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:00:48.185700 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.185626    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:00:48.189471 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:48.189442    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf356dcb1_1e78_4428_bc1f_43ede09e47e3.slice/crio-16dd33cae76bbd1b0c7cf9df21c54d531e8fb12ec67af2ef03b8cf6808763258 WatchSource:0}: Error finding container 16dd33cae76bbd1b0c7cf9df21c54d531e8fb12ec67af2ef03b8cf6808763258: Status 404 returned error can't find the container with id 16dd33cae76bbd1b0c7cf9df21c54d531e8fb12ec67af2ef03b8cf6808763258
Oct 30 21:00:48.191156 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.191033    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n8kk5"
Oct 30 21:00:48.200854 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.199822    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:00:48.200854 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:48.200777    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31769be6_0719_4533_997c_e8b458c54d79.slice/crio-9af15816b27b2d988ec634dff3db9c8da962fc18f5c0a7986a563729295d27ab WatchSource:0}: Error finding container 9af15816b27b2d988ec634dff3db9c8da962fc18f5c0a7986a563729295d27ab: Status 404 returned error can't find the container with id 9af15816b27b2d988ec634dff3db9c8da962fc18f5c0a7986a563729295d27ab
Oct 30 21:00:48.204671 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:48.204627    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829cdf21_2065_4e39_902d_7b857a2d74bd.slice/crio-dd8e2e8c7d57d31be34eb25e8e8c3e41bb9ff222a9310d7961bf75b4a9057e9a WatchSource:0}: Error finding container dd8e2e8c7d57d31be34eb25e8e8c3e41bb9ff222a9310d7961bf75b4a9057e9a: Status 404 returned error can't find the container with id dd8e2e8c7d57d31be34eb25e8e8c3e41bb9ff222a9310d7961bf75b4a9057e9a
Oct 30 21:00:48.209309 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:48.209276    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a08d3f6_f560_4507_9596_9b548981063c.slice/crio-678cfbf8b8db9480d32de5901202ec593769892267f0a058f6f6f843ddd9e7e4 WatchSource:0}: Error finding container 678cfbf8b8db9480d32de5901202ec593769892267f0a058f6f6f843ddd9e7e4: Status 404 returned error can't find the container with id 678cfbf8b8db9480d32de5901202ec593769892267f0a058f6f6f843ddd9e7e4
Oct 30 21:00:48.221584 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.221559    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:00:48.234135 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:00:48.234104    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d1bc79_bd2e_422f_b8aa_6de92eabb634.slice/crio-61ebb36595e21a7d9e24b5c1f205941d3f025ed75c8ff3b2b0ed16bcfc9b9bb1 WatchSource:0}: Error finding container 61ebb36595e21a7d9e24b5c1f205941d3f025ed75c8ff3b2b0ed16bcfc9b9bb1: Status 404 returned error can't find the container with id 61ebb36595e21a7d9e24b5c1f205941d3f025ed75c8ff3b2b0ed16bcfc9b9bb1
Oct 30 21:00:48.282107 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.282076    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:00:48.544847 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.544763    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:48.544847 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.544814    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:48.545056 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:48.544966    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:48.545056 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:48.545031    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs podName:d80303a1-1861-4958-a037-891078b48456 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:49.545013187 +0000 UTC m=+4.149957328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs") pod "network-metrics-daemon-xppxb" (UID: "d80303a1-1861-4958-a037-891078b48456") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:48.545485 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:48.545459    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:48.545625 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:48.545517    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret podName:b941fd2b-fbbd-4343-81e1-9a0cdef17a10 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:49.545502383 +0000 UTC m=+4.150446522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret") pod "global-pull-secret-syncer-qth9k" (UID: "b941fd2b-fbbd-4343-81e1-9a0cdef17a10") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:48.646013 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.645980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl567\" (UniqueName: \"kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567\") pod \"network-check-target-6slln\" (UID: \"684225a2-9b4e-4c1f-a784-85da7509bbf1\") " pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:48.646188 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:48.646141    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:00:48.646188 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:48.646163    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:00:48.646188 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:48.646175    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fl567 for pod openshift-network-diagnostics/network-check-target-6slln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:48.646353 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:48.646240    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567 podName:684225a2-9b4e-4c1f-a784-85da7509bbf1 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:49.646222262 +0000 UTC m=+4.251166401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fl567" (UniqueName: "kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567") pod "network-check-target-6slln" (UID: "684225a2-9b4e-4c1f-a784-85da7509bbf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:48.866625 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.866513    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:55:47 +0000 UTC" deadline="2028-07-20 20:52:37.694161382 +0000 UTC"
Oct 30 21:00:48.866625 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.866575    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15095h51m48.827590438s"
Oct 30 21:00:48.912726 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.912613    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n8kk5" event={"ID":"3a08d3f6-f560-4507-9596-9b548981063c","Type":"ContainerStarted","Data":"678cfbf8b8db9480d32de5901202ec593769892267f0a058f6f6f843ddd9e7e4"}
Oct 30 21:00:48.915654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.915541    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk" event={"ID":"31769be6-0719-4533-997c-e8b458c54d79","Type":"ContainerStarted","Data":"9af15816b27b2d988ec634dff3db9c8da962fc18f5c0a7986a563729295d27ab"}
Oct 30 21:00:48.922725 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.922648    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zbdld" event={"ID":"b459c3a5-a5d9-4776-9788-a7d7b943fa53","Type":"ContainerStarted","Data":"efb857e7821bb8b237f793852c65f05e1232206434296b808d46c55e506bb33f"}
Oct 30 21:00:48.927829 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.927750    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q7r7r" event={"ID":"082e6a92-3eef-423d-859e-405adb01c3a1","Type":"ContainerStarted","Data":"44cc001c54d126f5b3bad814f43e7bef6849925cecb8176809e742a8ae8ab773"}
Oct 30 21:00:48.934271 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.934193    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwgsc" event={"ID":"86d1bc79-bd2e-422f-b8aa-6de92eabb634","Type":"ContainerStarted","Data":"61ebb36595e21a7d9e24b5c1f205941d3f025ed75c8ff3b2b0ed16bcfc9b9bb1"}
Oct 30 21:00:48.937544 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.937485    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rzn4z" event={"ID":"829cdf21-2065-4e39-902d-7b857a2d74bd","Type":"ContainerStarted","Data":"dd8e2e8c7d57d31be34eb25e8e8c3e41bb9ff222a9310d7961bf75b4a9057e9a"}
Oct 30 21:00:48.941785 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.941757    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2z8lm" event={"ID":"f356dcb1-1e78-4428-bc1f-43ede09e47e3","Type":"ContainerStarted","Data":"16dd33cae76bbd1b0c7cf9df21c54d531e8fb12ec67af2ef03b8cf6808763258"}
Oct 30 21:00:48.953609 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.953577    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp" event={"ID":"98891442-5d51-4e57-a74a-f0fc22de1113","Type":"ContainerStarted","Data":"1933d1d2a61246b566d9781890abf1d4841a4f89cd7aa43ab6763d5b4f0005ac"}
Oct 30 21:00:48.955719 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:48.955647    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t48nt" event={"ID":"4e6f376c-f439-45e5-9049-d7d097391b83","Type":"ContainerStarted","Data":"a1a425e03ff21b01247dddb5563c5e2e576576d0c2ce995f5b8de71f7cd848df"}
Oct 30 21:00:49.554342 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:49.553329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:49.554342 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:49.553638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:49.554342 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:49.553801    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:49.554342 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:49.553868    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs podName:d80303a1-1861-4958-a037-891078b48456 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:51.553848988 +0000 UTC m=+6.158793122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs") pod "network-metrics-daemon-xppxb" (UID: "d80303a1-1861-4958-a037-891078b48456") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:49.554342 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:49.554254    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:49.554342 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:49.554301    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret podName:b941fd2b-fbbd-4343-81e1-9a0cdef17a10 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:51.5542865 +0000 UTC m=+6.159230631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret") pod "global-pull-secret-syncer-qth9k" (UID: "b941fd2b-fbbd-4343-81e1-9a0cdef17a10") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:49.655080 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:49.654452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl567\" (UniqueName: \"kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567\") pod \"network-check-target-6slln\" (UID: \"684225a2-9b4e-4c1f-a784-85da7509bbf1\") " pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:49.655080 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:49.654661    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:00:49.655080 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:49.654676    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:00:49.655080 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:49.654686    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fl567 for pod openshift-network-diagnostics/network-check-target-6slln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:49.655080 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:49.654743    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567 podName:684225a2-9b4e-4c1f-a784-85da7509bbf1 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:51.654726291 +0000 UTC m=+6.259670429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fl567" (UniqueName: "kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567") pod "network-check-target-6slln" (UID: "684225a2-9b4e-4c1f-a784-85da7509bbf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:49.867362 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:49.867318    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:55:47 +0000 UTC" deadline="2028-07-05 21:05:40.879570801 +0000 UTC"
Oct 30 21:00:49.867362 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:49.867359    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14736h4m51.012216362s"
Oct 30 21:00:49.907565 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:49.907526    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:49.907732 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:49.907673    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:00:49.908171 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:49.908149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:49.908286 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:49.908267    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:00:49.908456 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:49.908438    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:49.909033 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:49.909007    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:00:49.965455 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:49.965034    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"bb77e2dfdbc4a16f3bcede022ac475a7","Type":"ContainerStarted","Data":"d75bf0cbb543c4ff2bbe7a443dd1450b778b5b0717f28bbde7bfd8df08a8d4f1"}
Oct 30 21:00:49.969230 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:49.969204    2582 generic.go:356] "Generic (PLEG): container finished" podID="29fda1c39c64a6789fa2412aae915cee" containerID="7110cbf338332d0dc33d5e135e5562cb83974412cba79f94ec7ce4db02e8f8f2" exitCode=0
Oct 30 21:00:49.969362 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:49.969244    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"29fda1c39c64a6789fa2412aae915cee","Type":"ContainerDied","Data":"7110cbf338332d0dc33d5e135e5562cb83974412cba79f94ec7ce4db02e8f8f2"}
Oct 30 21:00:49.999104 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:49.998953    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" podStartSLOduration=2.9989329639999998 podStartE2EDuration="2.998932964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:49.980353808 +0000 UTC m=+4.585297948" watchObservedRunningTime="2026-10-30 21:00:49.998932964 +0000 UTC m=+4.603877110"
Oct 30 21:00:50.980858 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:50.980824    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"29fda1c39c64a6789fa2412aae915cee","Type":"ContainerStarted","Data":"581d5ba4cc3146cd3735936cc09bc51b72f841d7264067102629735592de0211"}
Oct 30 21:00:51.569280 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:51.568512    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:51.569280 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:51.568561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:51.569280 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:51.568703    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:51.569280 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:51.568763    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs podName:d80303a1-1861-4958-a037-891078b48456 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:55.568746315 +0000 UTC m=+10.173690437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs") pod "network-metrics-daemon-xppxb" (UID: "d80303a1-1861-4958-a037-891078b48456") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:51.569280 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:51.569143    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:51.569280 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:51.569188    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret podName:b941fd2b-fbbd-4343-81e1-9a0cdef17a10 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:55.56917364 +0000 UTC m=+10.174117763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret") pod "global-pull-secret-syncer-qth9k" (UID: "b941fd2b-fbbd-4343-81e1-9a0cdef17a10") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:51.669698 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:51.669662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl567\" (UniqueName: \"kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567\") pod \"network-check-target-6slln\" (UID: \"684225a2-9b4e-4c1f-a784-85da7509bbf1\") " pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:51.669885 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:51.669833    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:00:51.669885 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:51.669853    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:00:51.669885 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:51.669865    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fl567 for pod openshift-network-diagnostics/network-check-target-6slln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:51.670028 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:51.669918    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567 podName:684225a2-9b4e-4c1f-a784-85da7509bbf1 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:55.669901073 +0000 UTC m=+10.274845211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fl567" (UniqueName: "kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567") pod "network-check-target-6slln" (UID: "684225a2-9b4e-4c1f-a784-85da7509bbf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:51.897505 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:51.897422    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:51.897661 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:51.897555    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:00:51.897939 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:51.897920    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:51.898064 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:51.898035    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:00:51.898129 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:51.898083    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:51.898184 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:51.898145    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:00:51.990430 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:51.989667    2582 generic.go:356] "Generic (PLEG): container finished" podID="86d1bc79-bd2e-422f-b8aa-6de92eabb634" containerID="f3dff7f82613ac08dfb6881ab39e39ff157d8ed22a744dc354454c5c2d021907" exitCode=0
Oct 30 21:00:51.990430 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:51.989793    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwgsc" event={"ID":"86d1bc79-bd2e-422f-b8aa-6de92eabb634","Type":"ContainerDied","Data":"f3dff7f82613ac08dfb6881ab39e39ff157d8ed22a744dc354454c5c2d021907"}
Oct 30 21:00:52.008776 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:52.008721    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" podStartSLOduration=5.008702521 podStartE2EDuration="5.008702521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:50.993488367 +0000 UTC m=+5.598432507" watchObservedRunningTime="2026-10-30 21:00:52.008702521 +0000 UTC m=+6.613646661"
Oct 30 21:00:52.995854 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:52.995740    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t48nt" event={"ID":"4e6f376c-f439-45e5-9049-d7d097391b83","Type":"ContainerStarted","Data":"bdf7d9a9600c047bfe3a95a2663ff042ec317635689a62b0f5724d8194c888b8"}
Oct 30 21:00:53.897510 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:53.897476    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:53.897670 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:53.897606    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:00:53.900157 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:53.897808    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:53.900157 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:53.897935    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:00:53.900157 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:53.898007    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:53.900157 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:53.898090    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:00:54.002678 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:54.002483    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rzn4z" event={"ID":"829cdf21-2065-4e39-902d-7b857a2d74bd","Type":"ContainerStarted","Data":"dea437f659ee2d657dc907d32c76cb80f98f2efc1538d236353b397d66f826b4"}
Oct 30 21:00:54.018324 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:54.018272    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t48nt" podStartSLOduration=4.251039992 podStartE2EDuration="8.018252144s" totalImagesPullingTime="3.767212152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:53.018686433 +0000 UTC m=+7.623630588" watchObservedRunningTime="2026-10-30 21:00:54.018252144 +0000 UTC m=+8.623196285"
Oct 30 21:00:54.018608 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:54.018572    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rzn4z" podStartSLOduration=2.953503495 podStartE2EDuration="8.018561283s" totalImagesPullingTime="5.065057788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:54.017654844 +0000 UTC m=+8.622598986" watchObservedRunningTime="2026-10-30 21:00:54.018561283 +0000 UTC m=+8.623505424"
Oct 30 21:00:54.562670 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:54.562575    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:00:54.563015 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:54.563002    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:00:55.010490 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:55.009976    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp" event={"ID":"98891442-5d51-4e57-a74a-f0fc22de1113","Type":"ContainerStarted","Data":"bbb653807d09d9edd2e09ba6d60d97fc4d380575a6a4b1003b44f5975faf2b88"}
Oct 30 21:00:55.604408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:55.603627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:55.604408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:55.603681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:55.604408 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:55.603833    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:55.604408 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:55.603903    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs podName:d80303a1-1861-4958-a037-891078b48456 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:03.60388567 +0000 UTC m=+18.208829808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs") pod "network-metrics-daemon-xppxb" (UID: "d80303a1-1861-4958-a037-891078b48456") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:00:55.604408 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:55.604305    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:55.604408 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:55.604356    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret podName:b941fd2b-fbbd-4343-81e1-9a0cdef17a10 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:03.604341922 +0000 UTC m=+18.209286058 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret") pod "global-pull-secret-syncer-qth9k" (UID: "b941fd2b-fbbd-4343-81e1-9a0cdef17a10") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:00:55.704945 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:55.704228    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl567\" (UniqueName: \"kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567\") pod \"network-check-target-6slln\" (UID: \"684225a2-9b4e-4c1f-a784-85da7509bbf1\") " pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:55.704945 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:55.704485    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:00:55.704945 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:55.704508    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:00:55.704945 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:55.704525    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fl567 for pod openshift-network-diagnostics/network-check-target-6slln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:55.704945 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:55.704582    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567 podName:684225a2-9b4e-4c1f-a784-85da7509bbf1 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:03.70456412 +0000 UTC m=+18.309508259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fl567" (UniqueName: "kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567") pod "network-check-target-6slln" (UID: "684225a2-9b4e-4c1f-a784-85da7509bbf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:00:55.896337 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:55.895867    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:55.896337 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:55.895973    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:00:55.896337 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:55.896137    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:55.896337 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:55.896261    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:00:55.896337 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:55.896289    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:55.896747 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:55.896463    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:00:55.997710 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:55.997683    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:00:56.015293 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:56.015259    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp" event={"ID":"98891442-5d51-4e57-a74a-f0fc22de1113","Type":"ContainerStarted","Data":"e7b1cf0384c78d664833fbc298c77af831b0ec536028dbf25f162d889ceb4022"}
Oct 30 21:00:56.018174 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:56.018153    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:00:56.019176 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:56.019155    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n8kk5" event={"ID":"3a08d3f6-f560-4507-9596-9b548981063c","Type":"ContainerStarted","Data":"a1c1ca35c9e3194c43f5ca99b90e9adeee961ea2eb90b66397d72647242ac200"}
Oct 30 21:00:56.034435 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:56.034353    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-n8kk5" podStartSLOduration=2.407012936 podStartE2EDuration="10.034333227s" totalImagesPullingTime="7.627320291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:56.03282083 +0000 UTC m=+10.637765002" watchObservedRunningTime="2026-10-30 21:00:56.034333227 +0000 UTC m=+10.639277368"
Oct 30 21:00:56.889993 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:56.889883    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:00:55.997704602Z","UUID":"f4396856-b1bd-45d7-a303-d5328f8c7e1f","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:00:56.892051 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:56.892027    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:00:56.892234 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:56.892068    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:00:57.024300 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:57.024256    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q7r7r" event={"ID":"082e6a92-3eef-423d-859e-405adb01c3a1","Type":"ContainerStarted","Data":"9b4643491029e95faf517fb190691a7df1a6ae4a13f54ea7b97df870990a4ed4"}
Oct 30 21:00:57.026513 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:57.026489    2582 generic.go:356] "Generic (PLEG): container finished" podID="86d1bc79-bd2e-422f-b8aa-6de92eabb634" containerID="fd1412fdda0e22902a2491a6122b41eba79a0adc217c00ff16284f7dc0b77514" exitCode=0
Oct 30 21:00:57.026978 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:57.026961    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwgsc" event={"ID":"86d1bc79-bd2e-422f-b8aa-6de92eabb634","Type":"ContainerDied","Data":"fd1412fdda0e22902a2491a6122b41eba79a0adc217c00ff16284f7dc0b77514"}
Oct 30 21:00:57.052764 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:57.052717    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-q7r7r" podStartSLOduration=2.938887379 podStartE2EDuration="11.052702586s" totalImagesPullingTime="8.113815207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:57.037964818 +0000 UTC m=+11.642908959" watchObservedRunningTime="2026-10-30 21:00:57.052702586 +0000 UTC m=+11.657646716"
Oct 30 21:00:57.896103 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:57.895818    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:57.896258 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:57.896224    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:00:57.896733 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:57.895818    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:57.896733 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:57.896430    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:00:57.896733 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:57.895905    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:57.896733 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:57.896718    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:00:58.032305 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:58.032266    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp" event={"ID":"98891442-5d51-4e57-a74a-f0fc22de1113","Type":"ContainerStarted","Data":"602af0f012b4dbf8a3307892f09c6024a5ddf498e3f6ba9533eb743325a71ee1"}
Oct 30 21:00:58.034403 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:58.034345    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zbdld" event={"ID":"b459c3a5-a5d9-4776-9788-a7d7b943fa53","Type":"ContainerStarted","Data":"463e0901af3a7028221ebaaf18127d432e2b5ce0923e4c5ff4d06ebbb14dfba9"}
Oct 30 21:00:58.060790 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:58.059889    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp" podStartSLOduration=3.430270477 podStartE2EDuration="12.059872292s" totalImagesPullingTime="8.629601815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:58.047522248 +0000 UTC m=+12.652466400" watchObservedRunningTime="2026-10-30 21:00:58.059872292 +0000 UTC m=+12.664816432"
Oct 30 21:00:59.038268 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:59.038228    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2z8lm" event={"ID":"f356dcb1-1e78-4428-bc1f-43ede09e47e3","Type":"ContainerStarted","Data":"f7a08ae4d0e049977c1fabafca7a780e882474ae8a242e463ef0b0d49f43db18"}
Oct 30 21:00:59.041176 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:59.041115    2582 generic.go:356] "Generic (PLEG): container finished" podID="86d1bc79-bd2e-422f-b8aa-6de92eabb634" containerID="63b4db0ef787507c3f515f3416ef2c3b783fa50f3d73cdd851c9079f9057a7c6" exitCode=0
Oct 30 21:00:59.041300 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:59.041183    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwgsc" event={"ID":"86d1bc79-bd2e-422f-b8aa-6de92eabb634","Type":"ContainerDied","Data":"63b4db0ef787507c3f515f3416ef2c3b783fa50f3d73cdd851c9079f9057a7c6"}
Oct 30 21:00:59.052640 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:59.052589    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zbdld" podStartSLOduration=5.060426469 podStartE2EDuration="13.052572794s" totalImagesPullingTime="7.992146325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:58.059428381 +0000 UTC m=+12.664372522" watchObservedRunningTime="2026-10-30 21:00:59.052572794 +0000 UTC m=+13.657516935"
Oct 30 21:00:59.052942 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:59.052916    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2z8lm" podStartSLOduration=2.762423567 podStartE2EDuration="13.052909928s" totalImagesPullingTime="10.290486361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:59.052129979 +0000 UTC m=+13.657074121" watchObservedRunningTime="2026-10-30 21:00:59.052909928 +0000 UTC m=+13.657854069"
Oct 30 21:00:59.896119 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:59.895645    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:00:59.896119 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:59.895779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:00:59.896119 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:59.895871    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:00:59.896119 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:59.895772    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:00:59.896119 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:00:59.895652    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:00:59.896119 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:00:59.895954    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:01:00.042904 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:00.042881    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:01:00.043536 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:00.042991    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:01:00.044812 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:00.044590    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:01:01.047932 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:01.047894    2582 generic.go:356] "Generic (PLEG): container finished" podID="86d1bc79-bd2e-422f-b8aa-6de92eabb634" containerID="b931cd74696edf2560665870d5e0dffe0d25664866df3b90c2d52904d10f1985" exitCode=0
Oct 30 21:01:01.048648 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:01.047988    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwgsc" event={"ID":"86d1bc79-bd2e-422f-b8aa-6de92eabb634","Type":"ContainerDied","Data":"b931cd74696edf2560665870d5e0dffe0d25664866df3b90c2d52904d10f1985"}
Oct 30 21:01:01.895128 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:01.895096    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:01.895128 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:01.895118    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:01.895349 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:01.895236    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:01:01.895595 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:01.895581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:01.895697 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:01.895676    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:01:01.895821 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:01.895793    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:01:03.055172 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:03.055032    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk" event={"ID":"31769be6-0719-4533-997c-e8b458c54d79","Type":"ContainerStarted","Data":"5a8b1dca5db08af9c368d329ffe23e4431427d04ebfc785c94851f1bfc42ef52"}
Oct 30 21:01:03.055601 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:03.055215    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk" event={"ID":"31769be6-0719-4533-997c-e8b458c54d79","Type":"ContainerStarted","Data":"4bc129f8e81c4873083f15cb72c9594696d3790f171c1cfe06d0a5e27bdaf1bc"}
Oct 30 21:01:03.661798 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:03.661569    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:03.661798 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:03.661754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:03.661948 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:03.661711    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:01:03.661948 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:03.661845    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:01:03.661948 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:03.661863    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret podName:b941fd2b-fbbd-4343-81e1-9a0cdef17a10 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:19.661846639 +0000 UTC m=+34.266790777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret") pod "global-pull-secret-syncer-qth9k" (UID: "b941fd2b-fbbd-4343-81e1-9a0cdef17a10") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:01:03.661948 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:03.661882    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs podName:d80303a1-1861-4958-a037-891078b48456 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:19.661872107 +0000 UTC m=+34.266816241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs") pod "network-metrics-daemon-xppxb" (UID: "d80303a1-1861-4958-a037-891078b48456") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:01:03.762322 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:03.762275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl567\" (UniqueName: \"kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567\") pod \"network-check-target-6slln\" (UID: \"684225a2-9b4e-4c1f-a784-85da7509bbf1\") " pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:03.762498 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:03.762448    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:01:03.762498 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:03.762466    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:01:03.762498 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:03.762477    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fl567 for pod openshift-network-diagnostics/network-check-target-6slln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:01:03.762597 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:03.762525    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567 podName:684225a2-9b4e-4c1f-a784-85da7509bbf1 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:19.762512055 +0000 UTC m=+34.367456190 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fl567" (UniqueName: "kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567") pod "network-check-target-6slln" (UID: "684225a2-9b4e-4c1f-a784-85da7509bbf1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:01:03.895351 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:03.895316    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:03.895351 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:03.895345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:03.895556 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:03.895441    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:03.895587 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:03.895563    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:01:03.895689 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:03.895665    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:01:03.895746 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:03.895734    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:01:04.059448 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:04.059416    2582 generic.go:356] "Generic (PLEG): container finished" podID="86d1bc79-bd2e-422f-b8aa-6de92eabb634" containerID="317762c6211a62d02796b66e2b22665a79bdac82f492c517f02776be813d1f36" exitCode=0
Oct 30 21:01:04.059911 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:04.059504    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwgsc" event={"ID":"86d1bc79-bd2e-422f-b8aa-6de92eabb634","Type":"ContainerDied","Data":"317762c6211a62d02796b66e2b22665a79bdac82f492c517f02776be813d1f36"}
Oct 30 21:01:04.062419 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:04.062383    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk" event={"ID":"31769be6-0719-4533-997c-e8b458c54d79","Type":"ContainerStarted","Data":"29a19d1070e1d91ca458bbd030e1525535fe7ff9fbd9f7f10e5524980af62bdf"}
Oct 30 21:01:04.062463 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:04.062453    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk" event={"ID":"31769be6-0719-4533-997c-e8b458c54d79","Type":"ContainerStarted","Data":"effe5410da2e6add1fc4f0d0b5248aa51215d42a4d06a6082083aa04f29801de"}
Oct 30 21:01:04.062500 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:04.062487    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk" event={"ID":"31769be6-0719-4533-997c-e8b458c54d79","Type":"ContainerStarted","Data":"e90c34a73b20e6d271e2817fbb12da143828ba9d6e4ce13521010867374649b0"}
Oct 30 21:01:04.062529 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:04.062522    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk" event={"ID":"31769be6-0719-4533-997c-e8b458c54d79","Type":"ContainerStarted","Data":"a9cfca193956b64bf6225f9302f25dc7239f07066167ee32c74390a2363056ac"}
Oct 30 21:01:05.066627 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:05.066597    2582 generic.go:356] "Generic (PLEG): container finished" podID="86d1bc79-bd2e-422f-b8aa-6de92eabb634" containerID="98200d155179fed8f808f637d4568631b021e9d298732f18b2a295867fe71416" exitCode=0
Oct 30 21:01:05.067087 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:05.066637    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwgsc" event={"ID":"86d1bc79-bd2e-422f-b8aa-6de92eabb634","Type":"ContainerDied","Data":"98200d155179fed8f808f637d4568631b021e9d298732f18b2a295867fe71416"}
Oct 30 21:01:05.895753 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:05.895719    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:05.895927 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:05.895806    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:01:05.895927 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:05.895912    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:05.896044 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:05.896024    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:01:05.896121 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:05.896090    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:05.896191 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:05.896173    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:01:06.072154 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:06.072119    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lwgsc" event={"ID":"86d1bc79-bd2e-422f-b8aa-6de92eabb634","Type":"ContainerStarted","Data":"1a31169a663fab68026db74c99aa488b404b56fd2e00441b16207ac2c1734610"}
Oct 30 21:01:06.075295 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:06.075269    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk" event={"ID":"31769be6-0719-4533-997c-e8b458c54d79","Type":"ContainerStarted","Data":"5337d115eb2a36771265c32ef020cda38d00a830990e047bcc7f70950b9eeaab"}
Oct 30 21:01:06.093723 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:06.093680    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lwgsc" podStartSLOduration=8.490507287 podStartE2EDuration="20.093667344s" totalImagesPullingTime="11.603160057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:06.093131951 +0000 UTC m=+20.698076091" watchObservedRunningTime="2026-10-30 21:01:06.093667344 +0000 UTC m=+20.698611484"
Oct 30 21:01:07.895173 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:07.895140    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:07.895790 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:07.895252    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:01:07.895790 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:07.895140    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:07.895790 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:07.895290    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:07.895790 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:07.895341    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:01:07.895790 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:07.895439    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:01:08.082635 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:08.082599    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk" event={"ID":"31769be6-0719-4533-997c-e8b458c54d79","Type":"ContainerStarted","Data":"323e18bff29104411f2485c284e161b10c2c29a80f005ab787c25d1c96f52188"}
Oct 30 21:01:08.082879 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:08.082854    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:08.082958 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:08.082924    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:08.083010 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:08.082960    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:08.100130 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:08.099966    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:08.100253 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:08.100204    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:08.110541 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:08.110507    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk" podStartSLOduration=7.403228668 podStartE2EDuration="22.110495772s" totalImagesPullingTime="14.707267104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:08.109985877 +0000 UTC m=+22.714930016" watchObservedRunningTime="2026-10-30 21:01:08.110495772 +0000 UTC m=+22.715439912"
Oct 30 21:01:09.157270 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:09.157238    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xppxb"]
Oct 30 21:01:09.157782 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:09.157364    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:09.157782 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:09.157480    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:01:09.159637 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:09.159610    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qth9k"]
Oct 30 21:01:09.159891 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:09.159866    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:09.159989 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:09.159966    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:01:09.160421 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:09.160385    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6slln"]
Oct 30 21:01:09.160514 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:09.160501    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:09.160614 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:09.160590    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:01:10.895945 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:10.895910    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:10.895945 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:10.895946    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:10.896365 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:10.896052    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:10.896365 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:10.896060    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:01:10.896365 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:10.896123    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:01:10.896365 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:10.896181    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:01:12.895580 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:12.895546    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:12.895580 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:12.895572    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:12.895580 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:12.895589    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:12.896133 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:12.895674    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:01:12.896133 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:12.895789    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:01:12.896133 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:12.895860    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:01:14.895565 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:14.895535    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:14.895565 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:14.895564    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:14.896009 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:14.895635    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qth9k" podUID="b941fd2b-fbbd-4343-81e1-9a0cdef17a10"
Oct 30 21:01:14.896009 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:14.895697    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6slln" podUID="684225a2-9b4e-4c1f-a784-85da7509bbf1"
Oct 30 21:01:14.896009 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:14.895735    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:14.896009 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:14.895827    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xppxb" podUID="d80303a1-1861-4958-a037-891078b48456"
Oct 30 21:01:15.699947 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.699879    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeReady"
Oct 30 21:01:15.700059 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.699999    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:01:15.733517 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.733489    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-745kc"]
Oct 30 21:01:15.738859 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.738200    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fdn45"]
Oct 30 21:01:15.738859 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.738339    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-745kc"
Oct 30 21:01:15.743007 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.742981    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m2xqt"]
Oct 30 21:01:15.747714 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.747686    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 21:01:15.747909 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.747885    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fdn45"
Oct 30 21:01:15.748022 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.747918    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67bd499c65-7dz4d"]
Oct 30 21:01:15.748079 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.748032    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-m2xqt"
Oct 30 21:01:15.753162 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.753144    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"]
Oct 30 21:01:15.753285 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.753263    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.756792 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.756773    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6d5fbcf987-vhd7f"]
Oct 30 21:01:15.757106 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.757088    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:15.758812 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.758797    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 21:01:15.759050 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.759036    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 21:01:15.760096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.760080    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wg2wk"]
Oct 30 21:01:15.760211 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.760195    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.763433 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.762963    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-22hfw"]
Oct 30 21:01:15.763433 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.763084    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.766094 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.766077    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"]
Oct 30 21:01:15.766200 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.766158    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:15.770456 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.769375    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"]
Oct 30 21:01:15.770456 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.769977    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:15.774007 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.773858    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-22c7r"]
Oct 30 21:01:15.774007 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.773957    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:15.782172 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.782151    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"]
Oct 30 21:01:15.783102 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.783034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:15.786482 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.786449    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc"]
Oct 30 21:01:15.786586 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.786549    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:15.790870 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.790822    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67cc99cbd8-gdhf8"]
Oct 30 21:01:15.792986 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.792965    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc"
Oct 30 21:01:15.801803 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.801766    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8"]
Oct 30 21:01:15.803473 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.803453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.811866 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.811604    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hsrbr"]
Oct 30 21:01:15.812018 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.811936    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8"
Oct 30 21:01:15.820461 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.819364    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xm2j4"]
Oct 30 21:01:15.820461 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.819949    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:15.831037 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831013    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-22hfw"]
Oct 30 21:01:15.831147 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831056    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m2xqt"]
Oct 30 21:01:15.831147 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831078    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fdn45"]
Oct 30 21:01:15.831147 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831098    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc"]
Oct 30 21:01:15.831147 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831119    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"]
Oct 30 21:01:15.831147 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831136    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"]
Oct 30 21:01:15.831478 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831158    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8"]
Oct 30 21:01:15.831478 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831179    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-745kc"]
Oct 30 21:01:15.831478 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831199    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67bd499c65-7dz4d"]
Oct 30 21:01:15.831478 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831217    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wg2wk"]
Oct 30 21:01:15.831478 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831235    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"]
Oct 30 21:01:15.831478 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831258    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xm2j4"]
Oct 30 21:01:15.831478 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831276    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-22c7r"]
Oct 30 21:01:15.831478 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831299    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"]
Oct 30 21:01:15.831478 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831317    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67cc99cbd8-gdhf8"]
Oct 30 21:01:15.831478 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831334    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hsrbr"]
Oct 30 21:01:15.831478 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831360    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d5fbcf987-vhd7f"]
Oct 30 21:01:15.832017 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.831474    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xm2j4"
Oct 30 21:01:15.846441 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.846421    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 21:01:15.846574 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.846551    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:15.846678 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.846484    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:15.846678 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.846444    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:15.846678 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.846562    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 21:01:15.846678 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.846619    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 21:01:15.846678 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.846625    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:15.846678 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.846644    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:15.847014 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.846971    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:01:15.847014 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.846980    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:15.847014 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.846981    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:01:15.847165 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847022    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:01:15.847165 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847030    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 21:01:15.847165 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847036    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:15.847165 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847029    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:15.847165 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847051    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:01:15.847474 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847247    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 21:01:15.847474 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847261    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 21:01:15.847474 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847274    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:15.847474 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847281    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:01:15.847474 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847280    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:15.847474 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847296    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:01:15.847474 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847345    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:01:15.847872 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847861    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:01:15.847928 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847891    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:01:15.847928 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847906    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:01:15.848012 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847942    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:01:15.848012 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.847951    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:01:15.848140 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.848124    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 21:01:15.851432 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851410    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-stats-auth\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.851546 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851443    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-installation-pull-secrets\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.851546 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94385c98-9c6d-4fc1-9c48-38c9e5a95447-trusted-ca\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:15.851546 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:15.851703 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7df44\" (UniqueName: \"kubernetes.io/projected/3f89330a-b946-4b7d-8024-5ed5825719e3-kube-api-access-7df44\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:15.851703 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr66n\" (UniqueName: \"kubernetes.io/projected/a5946a39-1450-4d88-aae5-6f76b062fafb-kube-api-access-wr66n\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:15.851703 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851681    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-default-certificate\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.851842 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59d6f\" (UniqueName: \"kubernetes.io/projected/dccde73f-6cb5-467f-b650-48195b474db0-kube-api-access-59d6f\") pod \"kas-connection-checker-685895c7c9-745kc\" (UID: \"dccde73f-6cb5-467f-b650-48195b474db0\") " pod="kube-system/kas-connection-checker-685895c7c9-745kc"
Oct 30 21:01:15.851842 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-installation-pull-secrets\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.851952 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851883    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.851952 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851912    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d4a91f73-8795-4a9d-a1cc-55ed355c0312-snapshots\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.851952 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8jl8\" (UniqueName: \"kubernetes.io/projected/722bd8ce-7062-4e97-a0e8-0984c73c2128-kube-api-access-s8jl8\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:15.852089 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.851968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xfpv\" (UniqueName: \"kubernetes.io/projected/d4a91f73-8795-4a9d-a1cc-55ed355c0312-kube-api-access-4xfpv\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.852089 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.852027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a91f73-8795-4a9d-a1cc-55ed355c0312-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.852089 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.852054    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9p2b\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-kube-api-access-m9p2b\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.852240 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.852139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94385c98-9c6d-4fc1-9c48-38c9e5a95447-config\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:15.852240 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.852189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/722bd8ce-7062-4e97-a0e8-0984c73c2128-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:15.852240 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.852249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzw62\" (UniqueName: \"kubernetes.io/projected/72689c92-965b-4dfc-9f3c-4b68660f6e53-kube-api-access-xzw62\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:15.852573 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.852284    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4a91f73-8795-4a9d-a1cc-55ed355c0312-tmp\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.852573 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.852315    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4a91f73-8795-4a9d-a1cc-55ed355c0312-serving-cert\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.852773 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.852740    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 21:01:15.852849 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.852829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.852905 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.852893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-trusted-ca\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.852944 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.852933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb264\" (UniqueName: \"kubernetes.io/projected/7c27677e-bbe2-4457-b205-1faae8635cb0-kube-api-access-zb264\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.853005 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.852990    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a5946a39-1450-4d88-aae5-6f76b062fafb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:15.853045 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853017    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-certificates\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.853080 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853072    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jdg\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-kube-api-access-k9jdg\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.853112 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a91f73-8795-4a9d-a1cc-55ed355c0312-service-ca-bundle\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.853112 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853104    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/148c1862-34bb-4b75-af99-96c64f2058b8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:15.853189 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:15.853189 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853167    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm58p\" (UniqueName: \"kubernetes.io/projected/2c305cc1-5455-49d2-a20b-58b81ec38c3a-kube-api-access-hm58p\") pod \"volume-data-source-validator-58c9f5f46d-cwzsc\" (UID: \"2c305cc1-5455-49d2-a20b-58b81ec38c3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc"
Oct 30 21:01:15.853247 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f89330a-b946-4b7d-8024-5ed5825719e3-config\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:15.853290 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853258    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6tln\" (UniqueName: \"kubernetes.io/projected/94385c98-9c6d-4fc1-9c48-38c9e5a95447-kube-api-access-p6tln\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:15.853290 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-image-registry-private-configuration\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.853354 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6n55\" (UniqueName: \"kubernetes.io/projected/e8e758fb-aeed-4b68-ba87-664edc7423ae-kube-api-access-x6n55\") pod \"kas-connection-checker-685895c7c9-fdn45\" (UID: \"e8e758fb-aeed-4b68-ba87-664edc7423ae\") " pod="kube-system/kas-connection-checker-685895c7c9-fdn45"
Oct 30 21:01:15.853385 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853356    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-image-registry-private-configuration\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.853385 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853378    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94385c98-9c6d-4fc1-9c48-38c9e5a95447-serving-cert\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:15.853480 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-ca-trust-extracted\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.853480 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-certificates\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.853593 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56aaf442-dd68-4562-bc03-11ba5d35c28e-ca-trust-extracted\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.853593 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnbms\" (UniqueName: \"kubernetes.io/projected/14fc74c3-e4de-4506-a08b-f78dc63fe774-kube-api-access-fnbms\") pod \"kas-connection-checker-685895c7c9-m2xqt\" (UID: \"14fc74c3-e4de-4506-a08b-f78dc63fe774\") " pod="kube-system/kas-connection-checker-685895c7c9-m2xqt"
Oct 30 21:01:15.853704 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:15.853704 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/722bd8ce-7062-4e97-a0e8-0984c73c2128-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:15.853704 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.853877 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853708    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-trusted-ca\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.853877 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853760    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f89330a-b946-4b7d-8024-5ed5825719e3-serving-cert\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:15.853877 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-bound-sa-token\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.853877 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853866    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:15.854474 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.854474 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.853960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-bound-sa-token\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.856482 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.856466    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:01:15.856578 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.856479    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 21:01:15.856659 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.856646    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 21:01:15.858096 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858076    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 21:01:15.858231 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858121    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:01:15.858231 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858193    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:01:15.858231 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858222    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 21:01:15.858470 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858293    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 21:01:15.858470 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858293    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 21:01:15.858470 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858313    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 21:01:15.858470 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858329    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 21:01:15.858470 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858365    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 21:01:15.858828 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858538    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 21:01:15.858828 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858586    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 21:01:15.858828 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858608    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 21:01:15.858828 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858646    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5jrbn\""
Oct 30 21:01:15.858828 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858656    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gpwcl\""
Oct 30 21:01:15.858828 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858655    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dcwvc\""
Oct 30 21:01:15.858828 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858693    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6h7tv\""
Oct 30 21:01:15.858828 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858709    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gdckf\""
Oct 30 21:01:15.858828 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.858734    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9rr4h\""
Oct 30 21:01:15.859223 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.859209    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cp8br\""
Oct 30 21:01:15.859313 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.859248    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-s7wtw\""
Oct 30 21:01:15.859313 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.859252    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hz799\""
Oct 30 21:01:15.859313 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.859260    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hl7k8\""
Oct 30 21:01:15.859313 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.859272    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 21:01:15.859313 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.859279    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p98j7\""
Oct 30 21:01:15.859313 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.859307    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-6nsgx\""
Oct 30 21:01:15.859703 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.859421    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-d5xhz\""
Oct 30 21:01:15.954522 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.954450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:15.954522 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.954479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.954522 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.954496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-bound-sa-token\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.954522 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.954516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjgbs\" (UniqueName: \"kubernetes.io/projected/928df61d-6160-4586-acb9-d13cd90c1a72-kube-api-access-gjgbs\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:15.954994 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.954683    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:01:15.954994 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.954732    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-stats-auth\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.954994 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.954764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-installation-pull-secrets\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.954994 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.954776    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:16.454757151 +0000 UTC m=+31.059701289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : configmap references non-existent config key: service-ca.crt
Oct 30 21:01:15.954994 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.954810    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls podName:72689c92-965b-4dfc-9f3c-4b68660f6e53 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:16.454799818 +0000 UTC m=+31.059743940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fxrz8" (UID: "72689c92-965b-4dfc-9f3c-4b68660f6e53") : secret "samples-operator-tls" not found
Oct 30 21:01:15.954994 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.954825    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94385c98-9c6d-4fc1-9c48-38c9e5a95447-trusted-ca\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:15.954994 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.954855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:15.954994 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.954880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7df44\" (UniqueName: \"kubernetes.io/projected/3f89330a-b946-4b7d-8024-5ed5825719e3-kube-api-access-7df44\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:15.954994 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.954907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr66n\" (UniqueName: \"kubernetes.io/projected/a5946a39-1450-4d88-aae5-6f76b062fafb-kube-api-access-wr66n\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.955037    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-default-certificate\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59d6f\" (UniqueName: \"kubernetes.io/projected/dccde73f-6cb5-467f-b650-48195b474db0-kube-api-access-59d6f\") pod \"kas-connection-checker-685895c7c9-745kc\" (UID: \"dccde73f-6cb5-467f-b650-48195b474db0\") " pod="kube-system/kas-connection-checker-685895c7c9-745kc"
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.955111    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls podName:a5946a39-1450-4d88-aae5-6f76b062fafb nodeName:}" failed. No retries permitted until 2026-10-30 21:01:16.455093703 +0000 UTC m=+31.060037838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-shzqw" (UID: "a5946a39-1450-4d88-aae5-6f76b062fafb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955156    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-installation-pull-secrets\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955190    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/928df61d-6160-4586-acb9-d13cd90c1a72-config-volume\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955222    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d4a91f73-8795-4a9d-a1cc-55ed355c0312-snapshots\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8jl8\" (UniqueName: \"kubernetes.io/projected/722bd8ce-7062-4e97-a0e8-0984c73c2128-kube-api-access-s8jl8\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xfpv\" (UniqueName: \"kubernetes.io/projected/d4a91f73-8795-4a9d-a1cc-55ed355c0312-kube-api-access-4xfpv\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.955343    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.955360    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd499c65-7dz4d: secret "image-registry-tls" not found
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.955421    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls podName:56aaf442-dd68-4562-bc03-11ba5d35c28e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:16.45538625 +0000 UTC m=+31.060330384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls") pod "image-registry-67bd499c65-7dz4d" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e") : secret "image-registry-tls" not found
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955347    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a91f73-8795-4a9d-a1cc-55ed355c0312-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.955496 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955487    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9p2b\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-kube-api-access-m9p2b\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955518    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94385c98-9c6d-4fc1-9c48-38c9e5a95447-config\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/722bd8ce-7062-4e97-a0e8-0984c73c2128-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955580    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzw62\" (UniqueName: \"kubernetes.io/projected/72689c92-965b-4dfc-9f3c-4b68660f6e53-kube-api-access-xzw62\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955602    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4a91f73-8795-4a9d-a1cc-55ed355c0312-tmp\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955622    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4a91f73-8795-4a9d-a1cc-55ed355c0312-serving-cert\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955647    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjrwb\" (UniqueName: \"kubernetes.io/projected/c736b662-1f6a-4984-b82a-9d3b455ee18f-kube-api-access-zjrwb\") pod \"ingress-canary-xm2j4\" (UID: \"c736b662-1f6a-4984-b82a-9d3b455ee18f\") " pod="openshift-ingress-canary/ingress-canary-xm2j4"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-trusted-ca\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m7ht\" (UniqueName: \"kubernetes.io/projected/6e922433-5427-4340-b27e-cb16e9bd2d0a-kube-api-access-9m7ht\") pod \"network-check-source-6f78d59f99-fvdt8\" (UID: \"6e922433-5427-4340-b27e-cb16e9bd2d0a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955784    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94385c98-9c6d-4fc1-9c48-38c9e5a95447-trusted-ca\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb264\" (UniqueName: \"kubernetes.io/projected/7c27677e-bbe2-4457-b205-1faae8635cb0-kube-api-access-zb264\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955843    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a5946a39-1450-4d88-aae5-6f76b062fafb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-certificates\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jdg\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-kube-api-access-k9jdg\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a91f73-8795-4a9d-a1cc-55ed355c0312-service-ca-bundle\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.956215 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/148c1862-34bb-4b75-af99-96c64f2058b8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955971    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.955994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm58p\" (UniqueName: \"kubernetes.io/projected/2c305cc1-5455-49d2-a20b-58b81ec38c3a-kube-api-access-hm58p\") pod \"volume-data-source-validator-58c9f5f46d-cwzsc\" (UID: \"2c305cc1-5455-49d2-a20b-58b81ec38c3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f89330a-b946-4b7d-8024-5ed5825719e3-config\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6tln\" (UniqueName: \"kubernetes.io/projected/94385c98-9c6d-4fc1-9c48-38c9e5a95447-kube-api-access-p6tln\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956063    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-image-registry-private-configuration\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6n55\" (UniqueName: \"kubernetes.io/projected/e8e758fb-aeed-4b68-ba87-664edc7423ae-kube-api-access-x6n55\") pod \"kas-connection-checker-685895c7c9-fdn45\" (UID: \"e8e758fb-aeed-4b68-ba87-664edc7423ae\") " pod="kube-system/kas-connection-checker-685895c7c9-fdn45"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-image-registry-private-configuration\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94385c98-9c6d-4fc1-9c48-38c9e5a95447-serving-cert\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-ca-trust-extracted\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-certificates\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956184    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a91f73-8795-4a9d-a1cc-55ed355c0312-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56aaf442-dd68-4562-bc03-11ba5d35c28e-ca-trust-extracted\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnbms\" (UniqueName: \"kubernetes.io/projected/14fc74c3-e4de-4506-a08b-f78dc63fe774-kube-api-access-fnbms\") pod \"kas-connection-checker-685895c7c9-m2xqt\" (UID: \"14fc74c3-e4de-4506-a08b-f78dc63fe774\") " pod="kube-system/kas-connection-checker-685895c7c9-m2xqt"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:15.956934 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/722bd8ce-7062-4e97-a0e8-0984c73c2128-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956305    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c736b662-1f6a-4984-b82a-9d3b455ee18f-cert\") pod \"ingress-canary-xm2j4\" (UID: \"c736b662-1f6a-4984-b82a-9d3b455ee18f\") " pod="openshift-ingress-canary/ingress-canary-xm2j4"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956346    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928df61d-6160-4586-acb9-d13cd90c1a72-tmp-dir\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-trusted-ca\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f89330a-b946-4b7d-8024-5ed5825719e3-serving-cert\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956460    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-bound-sa-token\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.956707    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/722bd8ce-7062-4e97-a0e8-0984c73c2128-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.956789    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.956829    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:16.456815622 +0000 UTC m=+31.061759755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : secret "router-metrics-certs-default" not found
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.957126    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4a91f73-8795-4a9d-a1cc-55ed355c0312-tmp\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.957180    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d4a91f73-8795-4a9d-a1cc-55ed355c0312-snapshots\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.957278    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-certificates\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.957471    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a5946a39-1450-4d88-aae5-6f76b062fafb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:15.957654 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.957503    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-ca-trust-extracted\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.958373 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.958294    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a91f73-8795-4a9d-a1cc-55ed355c0312-service-ca-bundle\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.958373 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.958317    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-certificates\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.958499 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.958414    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:01:15.958499 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.958460    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert podName:148c1862-34bb-4b75-af99-96c64f2058b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:16.458445802 +0000 UTC m=+31.063389940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-22hfw" (UID: "148c1862-34bb-4b75-af99-96c64f2058b8") : secret "networking-console-plugin-cert" not found
Oct 30 21:01:15.958604 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.958524    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-trusted-ca\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.958656 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.958622    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:15.958656 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.958635    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc99cbd8-gdhf8: secret "image-registry-tls" not found
Oct 30 21:01:15.958756 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:15.958672    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls podName:ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:16.458659365 +0000 UTC m=+31.063603504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls") pod "image-registry-67cc99cbd8-gdhf8" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7") : secret "image-registry-tls" not found
Oct 30 21:01:15.959052 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.959027    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/148c1862-34bb-4b75-af99-96c64f2058b8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:15.959319 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.959297    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f89330a-b946-4b7d-8024-5ed5825719e3-config\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:15.959422 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.959327    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56aaf442-dd68-4562-bc03-11ba5d35c28e-ca-trust-extracted\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.960146 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.960120    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94385c98-9c6d-4fc1-9c48-38c9e5a95447-config\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:15.960222 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.960156    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-installation-pull-secrets\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.960795 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.960312    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4a91f73-8795-4a9d-a1cc-55ed355c0312-serving-cert\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.960795 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.960407    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-trusted-ca\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.960795 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.960607    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-stats-auth\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.961263 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.961238    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:15.961848 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.961822    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-installation-pull-secrets\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.961941 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.961918    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-default-certificate\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.962045 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.962024    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f89330a-b946-4b7d-8024-5ed5825719e3-serving-cert\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:15.963382 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.963358    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-image-registry-private-configuration\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.963382 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.963378    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-image-registry-private-configuration\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.963994 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.963970    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/722bd8ce-7062-4e97-a0e8-0984c73c2128-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:15.964180 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.964160    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94385c98-9c6d-4fc1-9c48-38c9e5a95447-serving-cert\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:15.967126 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.967107    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7df44\" (UniqueName: \"kubernetes.io/projected/3f89330a-b946-4b7d-8024-5ed5825719e3-kube-api-access-7df44\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:15.967514 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.967489    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-bound-sa-token\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.970134 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.969149    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm58p\" (UniqueName: \"kubernetes.io/projected/2c305cc1-5455-49d2-a20b-58b81ec38c3a-kube-api-access-hm58p\") pod \"volume-data-source-validator-58c9f5f46d-cwzsc\" (UID: \"2c305cc1-5455-49d2-a20b-58b81ec38c3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc"
Oct 30 21:01:15.970134 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.969810    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jdg\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-kube-api-access-k9jdg\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.970407 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.970360    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzw62\" (UniqueName: \"kubernetes.io/projected/72689c92-965b-4dfc-9f3c-4b68660f6e53-kube-api-access-xzw62\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:15.971879 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.971845    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8jl8\" (UniqueName: \"kubernetes.io/projected/722bd8ce-7062-4e97-a0e8-0984c73c2128-kube-api-access-s8jl8\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:15.971983 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.971956    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59d6f\" (UniqueName: \"kubernetes.io/projected/dccde73f-6cb5-467f-b650-48195b474db0-kube-api-access-59d6f\") pod \"kas-connection-checker-685895c7c9-745kc\" (UID: \"dccde73f-6cb5-467f-b650-48195b474db0\") " pod="kube-system/kas-connection-checker-685895c7c9-745kc"
Oct 30 21:01:15.972780 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.972759    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-bound-sa-token\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:15.972955 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.972917    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9p2b\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-kube-api-access-m9p2b\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:15.973252 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.973233    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6n55\" (UniqueName: \"kubernetes.io/projected/e8e758fb-aeed-4b68-ba87-664edc7423ae-kube-api-access-x6n55\") pod \"kas-connection-checker-685895c7c9-fdn45\" (UID: \"e8e758fb-aeed-4b68-ba87-664edc7423ae\") " pod="kube-system/kas-connection-checker-685895c7c9-fdn45"
Oct 30 21:01:15.973581 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.973560    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnbms\" (UniqueName: \"kubernetes.io/projected/14fc74c3-e4de-4506-a08b-f78dc63fe774-kube-api-access-fnbms\") pod \"kas-connection-checker-685895c7c9-m2xqt\" (UID: \"14fc74c3-e4de-4506-a08b-f78dc63fe774\") " pod="kube-system/kas-connection-checker-685895c7c9-m2xqt"
Oct 30 21:01:15.973711 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.973694    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb264\" (UniqueName: \"kubernetes.io/projected/7c27677e-bbe2-4457-b205-1faae8635cb0-kube-api-access-zb264\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:15.973748 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.973706    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xfpv\" (UniqueName: \"kubernetes.io/projected/d4a91f73-8795-4a9d-a1cc-55ed355c0312-kube-api-access-4xfpv\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:15.974023 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.974009    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr66n\" (UniqueName: \"kubernetes.io/projected/a5946a39-1450-4d88-aae5-6f76b062fafb-kube-api-access-wr66n\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:15.974936 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:15.974913    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6tln\" (UniqueName: \"kubernetes.io/projected/94385c98-9c6d-4fc1-9c48-38c9e5a95447-kube-api-access-p6tln\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:16.057843 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.057814    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c736b662-1f6a-4984-b82a-9d3b455ee18f-cert\") pod \"ingress-canary-xm2j4\" (UID: \"c736b662-1f6a-4984-b82a-9d3b455ee18f\") " pod="openshift-ingress-canary/ingress-canary-xm2j4"
Oct 30 21:01:16.057843 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.057845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:16.058060 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.057861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928df61d-6160-4586-acb9-d13cd90c1a72-tmp-dir\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:16.058060 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.057902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjgbs\" (UniqueName: \"kubernetes.io/projected/928df61d-6160-4586-acb9-d13cd90c1a72-kube-api-access-gjgbs\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:16.058060 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.057992    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:01:16.058060 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.058062    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls podName:928df61d-6160-4586-acb9-d13cd90c1a72 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:16.558043632 +0000 UTC m=+31.162987774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls") pod "dns-default-hsrbr" (UID: "928df61d-6160-4586-acb9-d13cd90c1a72") : secret "dns-default-metrics-tls" not found
Oct 30 21:01:16.058312 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.058100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/928df61d-6160-4586-acb9-d13cd90c1a72-config-volume\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:16.058312 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.058164    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjrwb\" (UniqueName: \"kubernetes.io/projected/c736b662-1f6a-4984-b82a-9d3b455ee18f-kube-api-access-zjrwb\") pod \"ingress-canary-xm2j4\" (UID: \"c736b662-1f6a-4984-b82a-9d3b455ee18f\") " pod="openshift-ingress-canary/ingress-canary-xm2j4"
Oct 30 21:01:16.058312 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.058198    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928df61d-6160-4586-acb9-d13cd90c1a72-tmp-dir\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:16.058312 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.058248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7ht\" (UniqueName: \"kubernetes.io/projected/6e922433-5427-4340-b27e-cb16e9bd2d0a-kube-api-access-9m7ht\") pod \"network-check-source-6f78d59f99-fvdt8\" (UID: \"6e922433-5427-4340-b27e-cb16e9bd2d0a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8"
Oct 30 21:01:16.058595 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.058578    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/928df61d-6160-4586-acb9-d13cd90c1a72-config-volume\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:16.060506 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.060486    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c736b662-1f6a-4984-b82a-9d3b455ee18f-cert\") pod \"ingress-canary-xm2j4\" (UID: \"c736b662-1f6a-4984-b82a-9d3b455ee18f\") " pod="openshift-ingress-canary/ingress-canary-xm2j4"
Oct 30 21:01:16.061268 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.061254    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-745kc"
Oct 30 21:01:16.067596 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.067578    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fdn45"
Oct 30 21:01:16.067941 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.067917    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjgbs\" (UniqueName: \"kubernetes.io/projected/928df61d-6160-4586-acb9-d13cd90c1a72-kube-api-access-gjgbs\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:16.068961 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.068943    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjrwb\" (UniqueName: \"kubernetes.io/projected/c736b662-1f6a-4984-b82a-9d3b455ee18f-kube-api-access-zjrwb\") pod \"ingress-canary-xm2j4\" (UID: \"c736b662-1f6a-4984-b82a-9d3b455ee18f\") " pod="openshift-ingress-canary/ingress-canary-xm2j4"
Oct 30 21:01:16.069084 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.069068    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m7ht\" (UniqueName: \"kubernetes.io/projected/6e922433-5427-4340-b27e-cb16e9bd2d0a-kube-api-access-9m7ht\") pod \"network-check-source-6f78d59f99-fvdt8\" (UID: \"6e922433-5427-4340-b27e-cb16e9bd2d0a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8"
Oct 30 21:01:16.074236 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.074180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-m2xqt"
Oct 30 21:01:16.129119 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.129068    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:16.146776 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.146683    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:16.160095 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.160071    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:16.165898 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.165761    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:16.170862 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.170844    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc"
Oct 30 21:01:16.183226 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.183025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8"
Oct 30 21:01:16.197171 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.197020    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xm2j4"
Oct 30 21:01:16.361880 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.361850    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fdn45"]
Oct 30 21:01:16.363448 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.363426    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-745kc"]
Oct 30 21:01:16.365496 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.365213    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-m2xqt"]
Oct 30 21:01:16.365935 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:01:16.365904    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e758fb_aeed_4b68_ba87_664edc7423ae.slice/crio-a3d8e9bfc84976c576ba9dff462d1ef51d4b6042e7dfa9418a04c46100bd55a4 WatchSource:0}: Error finding container a3d8e9bfc84976c576ba9dff462d1ef51d4b6042e7dfa9418a04c46100bd55a4: Status 404 returned error can't find the container with id a3d8e9bfc84976c576ba9dff462d1ef51d4b6042e7dfa9418a04c46100bd55a4
Oct 30 21:01:16.367790 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:01:16.367761    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccde73f_6cb5_467f_b650_48195b474db0.slice/crio-b34a7ed0210cebfb27a57648633e07b58a6eab52acd0df68b23753d21a94fab3 WatchSource:0}: Error finding container b34a7ed0210cebfb27a57648633e07b58a6eab52acd0df68b23753d21a94fab3: Status 404 returned error can't find the container with id b34a7ed0210cebfb27a57648633e07b58a6eab52acd0df68b23753d21a94fab3
Oct 30 21:01:16.368869 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:01:16.368846    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fc74c3_e4de_4506_a08b_f78dc63fe774.slice/crio-1f5cefb7c4ff81375bed60dd0b3aae60eb55bc8a4ba3ccf133b1618e1a0004c4 WatchSource:0}: Error finding container 1f5cefb7c4ff81375bed60dd0b3aae60eb55bc8a4ba3ccf133b1618e1a0004c4: Status 404 returned error can't find the container with id 1f5cefb7c4ff81375bed60dd0b3aae60eb55bc8a4ba3ccf133b1618e1a0004c4
Oct 30 21:01:16.427686 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.427652    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-wg2wk"]
Oct 30 21:01:16.432541 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:01:16.432509    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a91f73_8795_4a9d_a1cc_55ed355c0312.slice/crio-42898de6490228421f7a8f68d3f5652d8da719ae566852f79ba47fee9f997a8b WatchSource:0}: Error finding container 42898de6490228421f7a8f68d3f5652d8da719ae566852f79ba47fee9f997a8b: Status 404 returned error can't find the container with id 42898de6490228421f7a8f68d3f5652d8da719ae566852f79ba47fee9f997a8b
Oct 30 21:01:16.457167 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.454376    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"]
Oct 30 21:01:16.458289 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:01:16.458265    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722bd8ce_7062_4e97_a0e8_0984c73c2128.slice/crio-f5490caf648e02ca6cb15dee373ba2831458ac9635502f34e57224e88deda1c4 WatchSource:0}: Error finding container f5490caf648e02ca6cb15dee373ba2831458ac9635502f34e57224e88deda1c4: Status 404 returned error can't find the container with id f5490caf648e02ca6cb15dee373ba2831458ac9635502f34e57224e88deda1c4
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.462523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.462585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.462639    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.462671    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.462693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.462741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.462777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.462896    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.462908    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd499c65-7dz4d: secret "image-registry-tls" not found
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.462946    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls podName:56aaf442-dd68-4562-bc03-11ba5d35c28e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:17.462933296 +0000 UTC m=+32.067877429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls") pod "image-registry-67bd499c65-7dz4d" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e") : secret "image-registry-tls" not found
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.463013    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.463040    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:17.463032584 +0000 UTC m=+32.067976715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : secret "router-metrics-certs-default" not found
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.463073    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.463093    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert podName:148c1862-34bb-4b75-af99-96c64f2058b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:17.463086817 +0000 UTC m=+32.068030948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-22hfw" (UID: "148c1862-34bb-4b75-af99-96c64f2058b8") : secret "networking-console-plugin-cert" not found
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.463127    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:16.463141 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.463133    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc99cbd8-gdhf8: secret "image-registry-tls" not found
Oct 30 21:01:16.463950 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.463149    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls podName:ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:17.463143749 +0000 UTC m=+32.068087879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls") pod "image-registry-67cc99cbd8-gdhf8" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7") : secret "image-registry-tls" not found
Oct 30 21:01:16.463950 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.463181    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:01:16.463950 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.463200    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls podName:72689c92-965b-4dfc-9f3c-4b68660f6e53 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:17.463194766 +0000 UTC m=+32.068138897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fxrz8" (UID: "72689c92-965b-4dfc-9f3c-4b68660f6e53") : secret "samples-operator-tls" not found
Oct 30 21:01:16.463950 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.463239    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:17.463233401 +0000 UTC m=+32.068177533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : configmap references non-existent config key: service-ca.crt
Oct 30 21:01:16.463950 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.463317    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:01:16.463950 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.463342    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls podName:a5946a39-1450-4d88-aae5-6f76b062fafb nodeName:}" failed. No retries permitted until 2026-10-30 21:01:17.463335912 +0000 UTC m=+32.068280043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-shzqw" (UID: "a5946a39-1450-4d88-aae5-6f76b062fafb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:01:16.473817 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.473796    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"]
Oct 30 21:01:16.477090 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.476943    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-22c7r"]
Oct 30 21:01:16.479431 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:01:16.479407    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94385c98_9c6d_4fc1_9c48_38c9e5a95447.slice/crio-5360740e4869790164337c557c010af80d800091210296279f3c173bdc919b91 WatchSource:0}: Error finding container 5360740e4869790164337c557c010af80d800091210296279f3c173bdc919b91: Status 404 returned error can't find the container with id 5360740e4869790164337c557c010af80d800091210296279f3c173bdc919b91
Oct 30 21:01:16.491150 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.491125    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc"]
Oct 30 21:01:16.528179 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.528156    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8"]
Oct 30 21:01:16.529618 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.529601    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xm2j4"]
Oct 30 21:01:16.530797 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:01:16.530772    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc736b662_1f6a_4984_b82a_9d3b455ee18f.slice/crio-b8ddbcdd3b7cf9c7d341dab48ff46b3763030d3a8ec0920f8d1c231df8147978 WatchSource:0}: Error finding container b8ddbcdd3b7cf9c7d341dab48ff46b3763030d3a8ec0920f8d1c231df8147978: Status 404 returned error can't find the container with id b8ddbcdd3b7cf9c7d341dab48ff46b3763030d3a8ec0920f8d1c231df8147978
Oct 30 21:01:16.531336 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:01:16.531316    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e922433_5427_4340_b27e_cb16e9bd2d0a.slice/crio-17acf891410fa6976a8a2b36dd876108f6f3e4d53b230a7a5801b5deaf528ab5 WatchSource:0}: Error finding container 17acf891410fa6976a8a2b36dd876108f6f3e4d53b230a7a5801b5deaf528ab5: Status 404 returned error can't find the container with id 17acf891410fa6976a8a2b36dd876108f6f3e4d53b230a7a5801b5deaf528ab5
Oct 30 21:01:16.563246 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.563226    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:16.563375 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.563362    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:01:16.563451 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:16.563439    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls podName:928df61d-6160-4586-acb9-d13cd90c1a72 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:17.563420004 +0000 UTC m=+32.168364136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls") pod "dns-default-hsrbr" (UID: "928df61d-6160-4586-acb9-d13cd90c1a72") : secret "dns-default-metrics-tls" not found
Oct 30 21:01:16.895210 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.895180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:16.895352 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.895250    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:16.895352 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.895268    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:16.899227 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.899208    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:01:16.899359 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.899301    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qd7pl\""
Oct 30 21:01:16.900581 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.900563    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6bj8t\""
Oct 30 21:01:16.900698 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.900592    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6m5dr\""
Oct 30 21:01:16.900698 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:16.900595    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:01:17.109748 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.109704    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xm2j4" event={"ID":"c736b662-1f6a-4984-b82a-9d3b455ee18f","Type":"ContainerStarted","Data":"b8ddbcdd3b7cf9c7d341dab48ff46b3763030d3a8ec0920f8d1c231df8147978"}
Oct 30 21:01:17.111553 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.111521    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc" event={"ID":"2c305cc1-5455-49d2-a20b-58b81ec38c3a","Type":"ContainerStarted","Data":"cdf64675c597186b699eb6e8c513a134194d255b16512a48541911865a2d5a74"}
Oct 30 21:01:17.112914 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.112883    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r" event={"ID":"94385c98-9c6d-4fc1-9c48-38c9e5a95447","Type":"ContainerStarted","Data":"5360740e4869790164337c557c010af80d800091210296279f3c173bdc919b91"}
Oct 30 21:01:17.117546 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.117478    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c" event={"ID":"722bd8ce-7062-4e97-a0e8-0984c73c2128","Type":"ContainerStarted","Data":"f5490caf648e02ca6cb15dee373ba2831458ac9635502f34e57224e88deda1c4"}
Oct 30 21:01:17.120747 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.120713    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8" event={"ID":"6e922433-5427-4340-b27e-cb16e9bd2d0a","Type":"ContainerStarted","Data":"17acf891410fa6976a8a2b36dd876108f6f3e4d53b230a7a5801b5deaf528ab5"}
Oct 30 21:01:17.124291 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.124235    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8" event={"ID":"3f89330a-b946-4b7d-8024-5ed5825719e3","Type":"ContainerStarted","Data":"316b7db837bf8eb15989a93a351335526e85adefa4024525139cf16168c90991"}
Oct 30 21:01:17.126738 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.126700    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-m2xqt" event={"ID":"14fc74c3-e4de-4506-a08b-f78dc63fe774","Type":"ContainerStarted","Data":"dba2b472de40014cde82a9dbbcf3dac0ae7212fbf96b9bdcf7201aa1796dc4ad"}
Oct 30 21:01:17.126828 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.126750    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-m2xqt" event={"ID":"14fc74c3-e4de-4506-a08b-f78dc63fe774","Type":"ContainerStarted","Data":"1f5cefb7c4ff81375bed60dd0b3aae60eb55bc8a4ba3ccf133b1618e1a0004c4"}
Oct 30 21:01:17.130352 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.129600    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-745kc" event={"ID":"dccde73f-6cb5-467f-b650-48195b474db0","Type":"ContainerStarted","Data":"6b64d82f04bb2ff31b8d71e52770cb293df10703a05e55b1e6486e954b5238bc"}
Oct 30 21:01:17.130352 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.129660    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-745kc" event={"ID":"dccde73f-6cb5-467f-b650-48195b474db0","Type":"ContainerStarted","Data":"b34a7ed0210cebfb27a57648633e07b58a6eab52acd0df68b23753d21a94fab3"}
Oct 30 21:01:17.135228 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.134708    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wg2wk" event={"ID":"d4a91f73-8795-4a9d-a1cc-55ed355c0312","Type":"ContainerStarted","Data":"42898de6490228421f7a8f68d3f5652d8da719ae566852f79ba47fee9f997a8b"}
Oct 30 21:01:17.137548 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.137522    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fdn45" event={"ID":"e8e758fb-aeed-4b68-ba87-664edc7423ae","Type":"ContainerStarted","Data":"191e175730da0be23affa0f703e716ed531c6194560a156f5946ff033797ee41"}
Oct 30 21:01:17.137663 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.137565    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fdn45" event={"ID":"e8e758fb-aeed-4b68-ba87-664edc7423ae","Type":"ContainerStarted","Data":"a3d8e9bfc84976c576ba9dff462d1ef51d4b6042e7dfa9418a04c46100bd55a4"}
Oct 30 21:01:17.142495 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.142442    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-m2xqt" podStartSLOduration=285.142428379 podStartE2EDuration="4m45.142428379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:17.142059723 +0000 UTC m=+31.747003877" watchObservedRunningTime="2026-10-30 21:01:17.142428379 +0000 UTC m=+31.747372520"
Oct 30 21:01:17.173439 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.172149    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fdn45" podStartSLOduration=285.172135989 podStartE2EDuration="4m45.172135989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:17.171657453 +0000 UTC m=+31.776601606" watchObservedRunningTime="2026-10-30 21:01:17.172135989 +0000 UTC m=+31.777080126"
Oct 30 21:01:17.173439 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.172459    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-745kc" podStartSLOduration=285.172451935 podStartE2EDuration="4m45.172451935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:17.156547606 +0000 UTC m=+31.761491748" watchObservedRunningTime="2026-10-30 21:01:17.172451935 +0000 UTC m=+31.777396077"
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.473223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.473275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.473332    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.473369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.473448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.473509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.473566    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.473702    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.473716    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc99cbd8-gdhf8: secret "image-registry-tls" not found
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.473781    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls podName:ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:19.473756288 +0000 UTC m=+34.078700423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls") pod "image-registry-67cc99cbd8-gdhf8" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7") : secret "image-registry-tls" not found
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.474211    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.474262    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls podName:72689c92-965b-4dfc-9f3c-4b68660f6e53 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:19.474248653 +0000 UTC m=+34.079192786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fxrz8" (UID: "72689c92-965b-4dfc-9f3c-4b68660f6e53") : secret "samples-operator-tls" not found
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.474331    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:19.474321133 +0000 UTC m=+34.079265264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : configmap references non-existent config key: service-ca.crt
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.474384    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.474431    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls podName:a5946a39-1450-4d88-aae5-6f76b062fafb nodeName:}" failed. No retries permitted until 2026-10-30 21:01:19.474421616 +0000 UTC m=+34.079365749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-shzqw" (UID: "a5946a39-1450-4d88-aae5-6f76b062fafb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:01:17.474734 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.474489    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:17.475616 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.474500    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd499c65-7dz4d: secret "image-registry-tls" not found
Oct 30 21:01:17.475616 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.474529    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls podName:56aaf442-dd68-4562-bc03-11ba5d35c28e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:19.474519099 +0000 UTC m=+34.079463232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls") pod "image-registry-67bd499c65-7dz4d" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e") : secret "image-registry-tls" not found
Oct 30 21:01:17.475616 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.474581    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:01:17.475616 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.474607    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:19.474598362 +0000 UTC m=+34.079542495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : secret "router-metrics-certs-default" not found
Oct 30 21:01:17.475616 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.474655    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:01:17.475616 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.474685    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert podName:148c1862-34bb-4b75-af99-96c64f2058b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:19.47467463 +0000 UTC m=+34.079618766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-22hfw" (UID: "148c1862-34bb-4b75-af99-96c64f2058b8") : secret "networking-console-plugin-cert" not found
Oct 30 21:01:17.574889 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:17.574257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:17.574889 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.574496    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:01:17.574889 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:17.574555    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls podName:928df61d-6160-4586-acb9-d13cd90c1a72 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:19.574537871 +0000 UTC m=+34.179481989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls") pod "dns-default-hsrbr" (UID: "928df61d-6160-4586-acb9-d13cd90c1a72") : secret "dns-default-metrics-tls" not found
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.496755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.496805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.496865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.496897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.496942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.496991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.497030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497161    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497173    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc99cbd8-gdhf8: secret "image-registry-tls" not found
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497218    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls podName:ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:23.497206161 +0000 UTC m=+38.102150292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls") pod "image-registry-67cc99cbd8-gdhf8" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7") : secret "image-registry-tls" not found
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497556    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497577    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd499c65-7dz4d: secret "image-registry-tls" not found
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497592    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497604    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:23.497586282 +0000 UTC m=+38.102530416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : configmap references non-existent config key: service-ca.crt
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497661    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497688    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls podName:56aaf442-dd68-4562-bc03-11ba5d35c28e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:23.497670972 +0000 UTC m=+38.102615102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls") pod "image-registry-67bd499c65-7dz4d" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e") : secret "image-registry-tls" not found
Oct 30 21:01:19.497689 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497599    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:01:19.498950 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497706    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert podName:148c1862-34bb-4b75-af99-96c64f2058b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:23.497697073 +0000 UTC m=+38.102641201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-22hfw" (UID: "148c1862-34bb-4b75-af99-96c64f2058b8") : secret "networking-console-plugin-cert" not found
Oct 30 21:01:19.498950 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497662    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:01:19.498950 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497723    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls podName:72689c92-965b-4dfc-9f3c-4b68660f6e53 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:23.497715262 +0000 UTC m=+38.102659388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fxrz8" (UID: "72689c92-965b-4dfc-9f3c-4b68660f6e53") : secret "samples-operator-tls" not found
Oct 30 21:01:19.498950 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497740    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls podName:a5946a39-1450-4d88-aae5-6f76b062fafb nodeName:}" failed. No retries permitted until 2026-10-30 21:01:23.497731293 +0000 UTC m=+38.102675414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-shzqw" (UID: "a5946a39-1450-4d88-aae5-6f76b062fafb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:01:19.498950 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.497755    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:23.497747629 +0000 UTC m=+38.102691790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : secret "router-metrics-certs-default" not found
Oct 30 21:01:19.598983 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.598207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:19.598983 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.598564    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:01:19.598983 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.598628    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls podName:928df61d-6160-4586-acb9-d13cd90c1a72 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:23.598611404 +0000 UTC m=+38.203555545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls") pod "dns-default-hsrbr" (UID: "928df61d-6160-4586-acb9-d13cd90c1a72") : secret "dns-default-metrics-tls" not found
Oct 30 21:01:19.699928 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.699892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:19.700100 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.700041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:19.700244 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.700206    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:01:19.700628 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:19.700599    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs podName:d80303a1-1861-4958-a037-891078b48456 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:51.700561012 +0000 UTC m=+66.305505134 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs") pod "network-metrics-daemon-xppxb" (UID: "d80303a1-1861-4958-a037-891078b48456") : secret "metrics-daemon-secret" not found
Oct 30 21:01:19.703435 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.703412    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:19.801265 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.801063    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl567\" (UniqueName: \"kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567\") pod \"network-check-target-6slln\" (UID: \"684225a2-9b4e-4c1f-a784-85da7509bbf1\") " pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:19.805419 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.805345    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl567\" (UniqueName: \"kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567\") pod \"network-check-target-6slln\" (UID: \"684225a2-9b4e-4c1f-a784-85da7509bbf1\") " pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:19.953969 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.953934    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:19.972658 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:19.971444    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:20.158428 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:20.157670    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc" event={"ID":"2c305cc1-5455-49d2-a20b-58b81ec38c3a","Type":"ContainerStarted","Data":"2479db92368d17cafb30b04d4c4b16f59edacf2626cdabbe05aedc33ec21d375"}
Oct 30 21:01:20.173983 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:20.173919    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc" podStartSLOduration=254.601472181 podStartE2EDuration="4m17.173905534s" totalImagesPullingTime="2.572433353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:20.173347924 +0000 UTC m=+34.778292064" watchObservedRunningTime="2026-10-30 21:01:20.173905534 +0000 UTC m=+34.778849672"
Oct 30 21:01:20.290110 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:20.290026    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qth9k"]
Oct 30 21:01:20.297362 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:01:20.297315    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb941fd2b_fbbd_4343_81e1_9a0cdef17a10.slice/crio-cab1b668c0e5a4af90c1781a2128d9a4ad4e9280f8ca0cd8d2aef24101d1e7c6 WatchSource:0}: Error finding container cab1b668c0e5a4af90c1781a2128d9a4ad4e9280f8ca0cd8d2aef24101d1e7c6: Status 404 returned error can't find the container with id cab1b668c0e5a4af90c1781a2128d9a4ad4e9280f8ca0cd8d2aef24101d1e7c6
Oct 30 21:01:20.320365 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:20.320241    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6slln"]
Oct 30 21:01:20.324867 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:01:20.324836    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684225a2_9b4e_4c1f_a784_85da7509bbf1.slice/crio-1ee0ddfc4929dd3b2272af1892c0a9d186bf5a9ec3a383eb652460a2630fda74 WatchSource:0}: Error finding container 1ee0ddfc4929dd3b2272af1892c0a9d186bf5a9ec3a383eb652460a2630fda74: Status 404 returned error can't find the container with id 1ee0ddfc4929dd3b2272af1892c0a9d186bf5a9ec3a383eb652460a2630fda74
Oct 30 21:01:21.162721 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:21.162652    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6slln" event={"ID":"684225a2-9b4e-4c1f-a784-85da7509bbf1","Type":"ContainerStarted","Data":"1ee0ddfc4929dd3b2272af1892c0a9d186bf5a9ec3a383eb652460a2630fda74"}
Oct 30 21:01:21.164672 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:21.164644    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qth9k" event={"ID":"b941fd2b-fbbd-4343-81e1-9a0cdef17a10","Type":"ContainerStarted","Data":"cab1b668c0e5a4af90c1781a2128d9a4ad4e9280f8ca0cd8d2aef24101d1e7c6"}
Oct 30 21:01:22.172511 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.172475    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8" event={"ID":"3f89330a-b946-4b7d-8024-5ed5825719e3","Type":"ContainerStarted","Data":"7ca531b22ea9acafc81169267837908dce407429f090188af535124265eba246"}
Oct 30 21:01:22.175408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.175362    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wg2wk" event={"ID":"d4a91f73-8795-4a9d-a1cc-55ed355c0312","Type":"ContainerStarted","Data":"6e14fc5c841c51e63aa316c6956c7b8a70f535405321a4562b21b7f11df9b71c"}
Oct 30 21:01:22.178157 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.178135    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/0.log"
Oct 30 21:01:22.178291 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.178177    2582 generic.go:356] "Generic (PLEG): container finished" podID="94385c98-9c6d-4fc1-9c48-38c9e5a95447" containerID="56bc9e671fb44f93a8930aca647f4c53760397cff3692f8295bf69866e7e3127" exitCode=255
Oct 30 21:01:22.178384 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.178364    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r" event={"ID":"94385c98-9c6d-4fc1-9c48-38c9e5a95447","Type":"ContainerDied","Data":"56bc9e671fb44f93a8930aca647f4c53760397cff3692f8295bf69866e7e3127"}
Oct 30 21:01:22.178496 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.178483    2582 scope.go:122] "RemoveContainer" containerID="56bc9e671fb44f93a8930aca647f4c53760397cff3692f8295bf69866e7e3127"
Oct 30 21:01:22.180145 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.180076    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c" event={"ID":"722bd8ce-7062-4e97-a0e8-0984c73c2128","Type":"ContainerStarted","Data":"cd246e75275ecc57da3457502f755bca35aef04edd940fb4a38eaec779eb7c61"}
Oct 30 21:01:22.183507 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.183487    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ltjmm"]
Oct 30 21:01:22.203773 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.203704    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ltjmm"]
Oct 30 21:01:22.203951 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.203835    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.226052 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.225993    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8" podStartSLOduration=254.352447963 podStartE2EDuration="4m19.225975388s" totalImagesPullingTime="4.873527425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:22.201633673 +0000 UTC m=+36.806577814" watchObservedRunningTime="2026-10-30 21:01:22.225975388 +0000 UTC m=+36.830919529"
Oct 30 21:01:22.250000 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.249952    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c" podStartSLOduration=254.983032497 podStartE2EDuration="4m20.249936885s" totalImagesPullingTime="5.266904388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:22.248769364 +0000 UTC m=+36.853713529" watchObservedRunningTime="2026-10-30 21:01:22.249936885 +0000 UTC m=+36.854881025"
Oct 30 21:01:22.276241 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.276161    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-wg2wk" podStartSLOduration=255.375507966 podStartE2EDuration="4m20.276141207s" totalImagesPullingTime="4.900633241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:22.274548476 +0000 UTC m=+36.879492628" watchObservedRunningTime="2026-10-30 21:01:22.276141207 +0000 UTC m=+36.881085349"
Oct 30 21:01:22.290180 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.290154    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:01:22.300268 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.300215    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:01:22.300764 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.300646    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xpqzq\""
Oct 30 21:01:22.336281 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.336248    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.336411 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.336306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwd58\" (UniqueName: \"kubernetes.io/projected/72f357cb-3592-424c-9391-9058d0821d5e-kube-api-access-nwd58\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.336848 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.336825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72f357cb-3592-424c-9391-9058d0821d5e-crio-socket\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.336946 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.336931    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72f357cb-3592-424c-9391-9058d0821d5e-data-volume\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.336997 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.336960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.439165 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.437874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72f357cb-3592-424c-9391-9058d0821d5e-crio-socket\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.439165 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.437942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72f357cb-3592-424c-9391-9058d0821d5e-data-volume\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.439165 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.437969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.439165 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.438032    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.439165 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.438068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwd58\" (UniqueName: \"kubernetes.io/projected/72f357cb-3592-424c-9391-9058d0821d5e-kube-api-access-nwd58\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.439165 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.438225    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72f357cb-3592-424c-9391-9058d0821d5e-crio-socket\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.439165 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.438563    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72f357cb-3592-424c-9391-9058d0821d5e-data-volume\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.439165 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:22.438647    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:01:22.439165 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:22.438701    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:22.938681302 +0000 UTC m=+37.543625437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : configmap "kube-rbac-proxy" not found
Oct 30 21:01:22.439165 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:22.438808    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:22.439165 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:22.438844    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:22.938832999 +0000 UTC m=+37.543777121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:22.477488 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.477442    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwd58\" (UniqueName: \"kubernetes.io/projected/72f357cb-3592-424c-9391-9058d0821d5e-kube-api-access-nwd58\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.525619 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.524634    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh"]
Oct 30 21:01:22.558863 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.558833    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh"]
Oct 30 21:01:22.559017 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.558968    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh"
Oct 30 21:01:22.585612 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.563405    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lw2cs\""
Oct 30 21:01:22.610582 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.610540    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:22.610728 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.610632    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:01:22.641337 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.641306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qw2\" (UniqueName: \"kubernetes.io/projected/d12dc745-07dc-47b3-9658-a13cd5c86638-kube-api-access-r5qw2\") pod \"migrator-66df5d496b-fw8lh\" (UID: \"d12dc745-07dc-47b3-9658-a13cd5c86638\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh"
Oct 30 21:01:22.742835 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.742806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qw2\" (UniqueName: \"kubernetes.io/projected/d12dc745-07dc-47b3-9658-a13cd5c86638-kube-api-access-r5qw2\") pod \"migrator-66df5d496b-fw8lh\" (UID: \"d12dc745-07dc-47b3-9658-a13cd5c86638\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh"
Oct 30 21:01:22.752636 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.752581    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qw2\" (UniqueName: \"kubernetes.io/projected/d12dc745-07dc-47b3-9658-a13cd5c86638-kube-api-access-r5qw2\") pod \"migrator-66df5d496b-fw8lh\" (UID: \"d12dc745-07dc-47b3-9658-a13cd5c86638\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh"
Oct 30 21:01:22.906827 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.906790    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh"
Oct 30 21:01:22.944481 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.944451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.944600 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:22.944504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:22.944688 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:22.944675    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:22.944733 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:22.944725    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:23.944711879 +0000 UTC m=+38.549656011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:22.945019 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:22.945006    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:01:22.945058 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:22.945049    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:23.945036581 +0000 UTC m=+38.549980712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : configmap "kube-rbac-proxy" not found
Oct 30 21:01:23.187193 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.187111    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8" event={"ID":"6e922433-5427-4340-b27e-cb16e9bd2d0a","Type":"ContainerStarted","Data":"d4656c1240ea624211c920e4e31e318ac0319da97bd1f1dbc5d1daac2e660a56"}
Oct 30 21:01:23.189321 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.189258    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xm2j4" event={"ID":"c736b662-1f6a-4984-b82a-9d3b455ee18f","Type":"ContainerStarted","Data":"36fef5774f3be2c67dc39813da696844d36d591bdcdab330d82c9ff91b8b5b4a"}
Oct 30 21:01:23.191704 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.191680    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/1.log"
Oct 30 21:01:23.192213 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.192192    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/0.log"
Oct 30 21:01:23.192354 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.192234    2582 generic.go:356] "Generic (PLEG): container finished" podID="94385c98-9c6d-4fc1-9c48-38c9e5a95447" containerID="6208c4ec3241c702493a4d785547fbe681ffe33610f047305caa3bea8d1962b1" exitCode=255
Oct 30 21:01:23.192354 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.192309    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r" event={"ID":"94385c98-9c6d-4fc1-9c48-38c9e5a95447","Type":"ContainerDied","Data":"6208c4ec3241c702493a4d785547fbe681ffe33610f047305caa3bea8d1962b1"}
Oct 30 21:01:23.192354 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.192347    2582 scope.go:122] "RemoveContainer" containerID="56bc9e671fb44f93a8930aca647f4c53760397cff3692f8295bf69866e7e3127"
Oct 30 21:01:23.192596 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.192559    2582 scope.go:122] "RemoveContainer" containerID="6208c4ec3241c702493a4d785547fbe681ffe33610f047305caa3bea8d1962b1"
Oct 30 21:01:23.192796 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.192751    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-22c7r_openshift-console-operator(94385c98-9c6d-4fc1-9c48-38c9e5a95447)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r" podUID="94385c98-9c6d-4fc1-9c48-38c9e5a95447"
Oct 30 21:01:23.204713 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.204670    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8" podStartSLOduration=254.476013567 podStartE2EDuration="4m20.204656127s" totalImagesPullingTime="5.72864256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:23.203551073 +0000 UTC m=+37.808495238" watchObservedRunningTime="2026-10-30 21:01:23.204656127 +0000 UTC m=+37.809600267"
Oct 30 21:01:23.218766 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.218746    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh"]
Oct 30 21:01:23.220761 ip-10-0-139-90 kubenswrapper[2582]: W1030 21:01:23.220732    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12dc745_07dc_47b3_9658_a13cd5c86638.slice/crio-cf5983c6730cb3c317d46989adcdfd951c86a63a57c8deda9ba3131a24d1bb28 WatchSource:0}: Error finding container cf5983c6730cb3c317d46989adcdfd951c86a63a57c8deda9ba3131a24d1bb28: Status 404 returned error can't find the container with id cf5983c6730cb3c317d46989adcdfd951c86a63a57c8deda9ba3131a24d1bb28
Oct 30 21:01:23.242281 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.242221    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xm2j4" podStartSLOduration=2.518734801 podStartE2EDuration="8.242202919s" totalImagesPullingTime="5.723468118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:23.240582374 +0000 UTC m=+37.845526525" watchObservedRunningTime="2026-10-30 21:01:23.242202919 +0000 UTC m=+37.847147061"
Oct 30 21:01:23.552725 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.552687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:23.552918 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.552740    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:23.552918 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.552789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:23.552918 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.552801    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:01:23.552918 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.552826    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:23.552918 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.552864    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls podName:72689c92-965b-4dfc-9f3c-4b68660f6e53 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.552846422 +0000 UTC m=+46.157790564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fxrz8" (UID: "72689c92-965b-4dfc-9f3c-4b68660f6e53") : secret "samples-operator-tls" not found
Oct 30 21:01:23.552918 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.552909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:23.552918 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.552918    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:01:23.553271 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.552939    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.552917467 +0000 UTC m=+46.157861608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : configmap references non-existent config key: service-ca.crt
Oct 30 21:01:23.553271 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.552962    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:01:23.553271 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.552995    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls podName:a5946a39-1450-4d88-aae5-6f76b062fafb nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.552985122 +0000 UTC m=+46.157929243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-shzqw" (UID: "a5946a39-1450-4d88-aae5-6f76b062fafb") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:01:23.553271 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.553012    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.553002342 +0000 UTC m=+46.157946460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : secret "router-metrics-certs-default" not found
Oct 30 21:01:23.553271 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.553074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:23.553271 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.553118    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:23.553271 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.553146    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd499c65-7dz4d: secret "image-registry-tls" not found
Oct 30 21:01:23.553271 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.553186    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:23.553271 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.553195    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls podName:56aaf442-dd68-4562-bc03-11ba5d35c28e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.553185509 +0000 UTC m=+46.158129640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls") pod "image-registry-67bd499c65-7dz4d" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e") : secret "image-registry-tls" not found
Oct 30 21:01:23.553271 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.553197    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc99cbd8-gdhf8: secret "image-registry-tls" not found
Oct 30 21:01:23.553271 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.553127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:23.553271 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.553229    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls podName:ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.553219768 +0000 UTC m=+46.158163908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls") pod "image-registry-67cc99cbd8-gdhf8" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7") : secret "image-registry-tls" not found
Oct 30 21:01:23.553855 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.553275    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:01:23.553855 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.553355    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert podName:148c1862-34bb-4b75-af99-96c64f2058b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.553338055 +0000 UTC m=+46.158282187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-22hfw" (UID: "148c1862-34bb-4b75-af99-96c64f2058b8") : secret "networking-console-plugin-cert" not found
Oct 30 21:01:23.654286 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.654257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:23.654458 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.654377    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:01:23.654458 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.654455    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls podName:928df61d-6160-4586-acb9-d13cd90c1a72 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.654442234 +0000 UTC m=+46.259386370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls") pod "dns-default-hsrbr" (UID: "928df61d-6160-4586-acb9-d13cd90c1a72") : secret "dns-default-metrics-tls" not found
Oct 30 21:01:23.957219 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.957189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:23.957339 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:23.957274    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:23.957541 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.957524    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:23.957593 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.957584    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:25.957566684 +0000 UTC m=+40.562510801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:23.957639 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.957634    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:01:23.957677 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:23.957667    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:25.957656218 +0000 UTC m=+40.562600338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : configmap "kube-rbac-proxy" not found
Oct 30 21:01:24.198408 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:24.198300    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6slln" event={"ID":"684225a2-9b4e-4c1f-a784-85da7509bbf1","Type":"ContainerStarted","Data":"4f907859c47d7cef3bed26f1e6f1cebf07e06714d19715f476b413a0927050ff"}
Oct 30 21:01:24.198864 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:24.198425    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:24.200040 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:24.200012    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh" event={"ID":"d12dc745-07dc-47b3-9658-a13cd5c86638","Type":"ContainerStarted","Data":"cf5983c6730cb3c317d46989adcdfd951c86a63a57c8deda9ba3131a24d1bb28"}
Oct 30 21:01:24.202011 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:24.201958    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/1.log"
Oct 30 21:01:24.202690 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:24.202671    2582 scope.go:122] "RemoveContainer" containerID="6208c4ec3241c702493a4d785547fbe681ffe33610f047305caa3bea8d1962b1"
Oct 30 21:01:24.202874 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:24.202850    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-22c7r_openshift-console-operator(94385c98-9c6d-4fc1-9c48-38c9e5a95447)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r" podUID="94385c98-9c6d-4fc1-9c48-38c9e5a95447"
Oct 30 21:01:24.217189 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:24.217146    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6slln" podStartSLOduration=34.694004535 podStartE2EDuration="38.217135271s" totalImagesPullingTime="3.523130736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:24.215246817 +0000 UTC m=+38.820190961" watchObservedRunningTime="2026-10-30 21:01:24.217135271 +0000 UTC m=+38.822079410"
Oct 30 21:01:25.207493 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:25.207406    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh" event={"ID":"d12dc745-07dc-47b3-9658-a13cd5c86638","Type":"ContainerStarted","Data":"93cd9baa523453436303dc034d894a1f76e8204752e147f60f637738fba8aa7b"}
Oct 30 21:01:25.223814 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:25.223760    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh" podStartSLOduration=1.514634349 podStartE2EDuration="3.223747148s" totalImagesPullingTime="1.709112799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:25.221851641 +0000 UTC m=+39.826795811" watchObservedRunningTime="2026-10-30 21:01:25.223747148 +0000 UTC m=+39.828691288"
Oct 30 21:01:25.978514 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:25.978476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:25.978700 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:25.978629    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:25.978700 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:25.978688    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:25.978815 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:25.978780    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:29.978761802 +0000 UTC m=+44.583705937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:25.978815 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:25.978801    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:01:25.978936 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:25.978921    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:29.978904901 +0000 UTC m=+44.583849026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : configmap "kube-rbac-proxy" not found
Oct 30 21:01:26.161271 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:26.161240    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:26.161488 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:26.161312    2582 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:26.161796 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:26.161779    2582 scope.go:122] "RemoveContainer" containerID="6208c4ec3241c702493a4d785547fbe681ffe33610f047305caa3bea8d1962b1"
Oct 30 21:01:26.162029 ip-10-0-139-90 kubenswrapper[2582]: E1030 21:01:26.161998    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-22c7r_openshift-console-operator(94385c98-9c6d-4fc1-9c48-38c9e5a95447)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r" podUID="94385c98-9c6d-4fc1-9c48-38c9e5a95447"
Oct 30 21:01:26.899442 ip-10-0-139-90 kubenswrapper[2582]: I1030 21:01:26.899301    2582 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:01:26.899477 ip-10-0-139-90 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:01:26.909831 ip-10-0-139-90 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:01:26.909994 ip-10-0-139-90 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:01:26.910042 ip-10-0-139-90 systemd[1]: kubelet.service: Consumed 3.020s CPU time.
Oct 30 21:01:26.921959 ip-10-0-139-90 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:01:26.983614 ip-10-0-139-90 kubenswrapper[4918]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:01:26.983614 ip-10-0-139-90 kubenswrapper[4918]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:01:26.983614 ip-10-0-139-90 kubenswrapper[4918]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:01:26.983614 ip-10-0-139-90 kubenswrapper[4918]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:01:26.985672 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985595    4918 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:26.985672 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985671    4918 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:26.985672 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985675    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985678    4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985681    4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985684    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985686    4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985690    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985693    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985695    4918 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985698    4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985700    4918 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985703    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985706    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985708    4918 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985712    4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985716    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985719    4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985722    4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985725    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985727    4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:26.985779 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985730    4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985732    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985735    4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985738    4918 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985741    4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985743    4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985753    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985756    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985758    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985761    4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985763    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985766    4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985778    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985781    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985783    4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985787    4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985789    4918 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985792    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985795    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:26.986272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985798    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985800    4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985803    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985805    4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985807    4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985810    4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985813    4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985815    4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985818    4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985821    4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985824    4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985826    4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985829    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985831    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985834    4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985837    4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985840    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985843    4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985845    4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985848    4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:26.986776 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985850    4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985852    4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985855    4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985857    4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985860    4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985862    4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985864    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985867    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985870    4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985873    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985876    4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985878    4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985881    4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985883    4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985886    4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985888    4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985890    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985893    4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985896    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985898    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:26.987272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985901    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985903    4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985905    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985908    4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985910    4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985921    4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985923    4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985927    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985930    4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985932    4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985935    4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985937    4918 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985940    4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985942    4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985948    4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985951    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985953    4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985956    4918 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985959    4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985961    4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:26.987823 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985964    4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985968    4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985972    4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985976    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985978    4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.985981    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986742    4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986749    4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986751    4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986754    4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986757    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986759    4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986762    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986765    4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986769    4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986773    4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986777    4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986780    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:26.988334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986786    4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986789    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986792    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986794    4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986796    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986799    4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986802    4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986804    4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986807    4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986810    4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986812    4918 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986815    4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986817    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986819    4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986822    4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986824    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986827    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986829    4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986831    4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986834    4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:26.988809 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986836    4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986839    4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986841    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986843    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986846    4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986848    4918 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986851    4918 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986854    4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986857    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986859    4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986861    4918 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986864    4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986866    4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986869    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986872    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986874    4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986876    4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986879    4918 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986882    4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986884    4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:26.989299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986886    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986889    4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986891    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986894    4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986896    4918 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986899    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986901    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986903    4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986906    4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986908    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986910    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986913    4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986915    4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986918    4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986921    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986923    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986925    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986928    4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986931    4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986933    4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:26.989849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986936    4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986938    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986940    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986943    4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986948    4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986952    4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986961    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986965    4918 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986967    4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986970    4918 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986973    4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986975    4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986978    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986980    4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986983    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986985    4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986987    4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986990    4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986992    4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:26.990352 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986995    4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.986997    4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987000    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987002    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987004    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987007    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987009    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987012    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987014    4918 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987017    4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987020    4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987022    4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987025    4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987028    4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987031    4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987638    4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987646    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987649    4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987653    4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987663    4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:26.990833 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987666    4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987669    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987671    4918 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987674    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987677    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987679    4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987681    4918 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987684    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987687    4918 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987689    4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987691    4918 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987694    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987696    4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987699    4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987701    4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987703    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987706    4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987709    4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987712    4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987714    4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:26.991334 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987717    4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987719    4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987722    4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987724    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987726    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987731    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987734    4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987736    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987739    4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987741    4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987744    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987746    4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987754    4918 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987757    4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987760    4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987762    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987765    4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987767    4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987769    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:26.991880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987772    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987774    4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987777    4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987779    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987782    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987784    4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987787    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987789    4918 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987792    4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987795    4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987798    4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987800    4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987802    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987805    4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987807    4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987810    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987812    4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987815    4918 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987819    4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987821    4918 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:26.992701 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987824    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987826    4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987829    4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987832    4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987834    4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987837    4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987845    4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987847    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987852    4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987856    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987859    4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987862    4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987865    4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987867    4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987871    4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987875    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987881    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987884    4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987887    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:26.993284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987889    4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987892    4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987894    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987896    4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987899    4918 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987901    4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987904    4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987906    4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987909    4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987911    4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987913    4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987917    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987919    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987922    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987925    4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987927    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987930    4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987933    4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987936    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:26.993945 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987938    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987944    4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987947    4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.987949    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988050    4918 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988069    4918 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988078    4918 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988082    4918 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988087    4918 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988090    4918 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988095    4918 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988098    4918 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988102    4918 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988104    4918 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988108    4918 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988111    4918 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988114    4918 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988117    4918 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988120    4918 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988123    4918 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988126    4918 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988129    4918 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988133    4918 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988136    4918 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:01:26.994460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988140    4918 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988144    4918 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988147    4918 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988151    4918 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988154    4918 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988157    4918 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988160    4918 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988163    4918 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988166    4918 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988170    4918 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988174    4918 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988177    4918 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988181    4918 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988184    4918 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988186    4918 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988189    4918 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988192    4918 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988195    4918 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988199    4918 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988202    4918 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988205    4918 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988208    4918 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988211    4918 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988215    4918 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:01:26.995047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988218    4918 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988221    4918 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988224    4918 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988227    4918 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988230    4918 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988233    4918 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988235    4918 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988238    4918 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988243    4918 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988246    4918 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988250    4918 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988253    4918 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988256    4918 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988259    4918 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988262    4918 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988265    4918 flags.go:64] FLAG: --help="false"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988268    4918 flags.go:64] FLAG: --hostname-override="ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988272    4918 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988275    4918 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988279    4918 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988282    4918 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988285    4918 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988288    4918 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:01:26.995691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988290    4918 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988293    4918 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988313    4918 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988316    4918 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988319    4918 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988323    4918 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988326    4918 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988329    4918 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988331    4918 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988334    4918 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988337    4918 flags.go:64] FLAG: --lock-file=""
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988340    4918 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988343    4918 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988346    4918 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988351    4918 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988354    4918 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988358    4918 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988361    4918 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988364    4918 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988367    4918 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988370    4918 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988373    4918 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988378    4918 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988381    4918 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988403    4918 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:01:26.996263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988406    4918 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988410    4918 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988413    4918 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988416    4918 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988419    4918 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988422    4918 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988425    4918 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988432    4918 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988435    4918 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988438    4918 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988441    4918 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988444    4918 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988446    4918 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988449    4918 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988452    4918 flags.go:64] FLAG: --port="10250"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988455    4918 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988458    4918 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-004282084a930176a"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988461    4918 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988464    4918 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988466    4918 flags.go:64] FLAG: --register-node="true"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988469    4918 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988473    4918 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988475    4918 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988478    4918 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988482    4918 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988486    4918 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:01:26.996945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988489    4918 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988492    4918 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988495    4918 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988498    4918 flags.go:64] FLAG: --runonce="false"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988500    4918 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988503    4918 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988506    4918 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988511    4918 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988514    4918 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988517    4918 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988520    4918 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988523    4918 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988527    4918 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988530    4918 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988532    4918 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988535    4918 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988538    4918 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988541    4918 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988544    4918 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988549    4918 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988551    4918 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988554    4918 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988559    4918 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988561    4918 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988564    4918 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:01:26.997678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988567    4918 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988570    4918 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988573    4918 flags.go:64] FLAG: --v="2"
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988577    4918 flags.go:64] FLAG: --version="false"
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988581    4918 flags.go:64] FLAG: --vmodule=""
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988586    4918 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.988589    4918 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988753    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988758    4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988762    4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988766    4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988769    4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988772    4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988775    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988781    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988783    4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988786    4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988788    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988791    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988794    4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988797    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:26.998373 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988799    4918 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988802    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988804    4918 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988807    4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988810    4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988812    4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988815    4918 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988817    4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988820    4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988823    4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988825    4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988828    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988830    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988833    4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988835    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988838    4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988841    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988844    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988846    4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988848    4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:26.999040 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988851    4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988853    4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988855    4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988858    4918 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988860    4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988864    4918 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988867    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988869    4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988872    4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988874    4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988876    4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988879    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988882    4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988884    4918 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988886    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988889    4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988891    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988894    4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988896    4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988899    4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:26.999592 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988901    4918 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988903    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988906    4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988908    4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988910    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988913    4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988915    4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988917    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988920    4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988923    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988926    4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988928    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988931    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988933    4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988936    4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988940    4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988943    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988946    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988949    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:27.000138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988951    4918 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988954    4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988956    4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988958    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988961    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988964    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988966    4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988968    4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988971    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988973    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988976    4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988978    4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988981    4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988983    4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988985    4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988988    4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988990    4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988993    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988995    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:27.000619 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.988998    4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989000    4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989003    4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989005    4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989008    4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989010    4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989013    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989015    4918 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989018    4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989020    4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989023    4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989027    4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989029    4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989031    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.989057    4918 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989092    4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:27.001140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989096    4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989099    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989102    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989105    4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989109    4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989113    4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989115    4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989118    4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989121    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989124    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989126    4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989129    4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989131    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989133    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989136    4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989138    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989142    4918 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989145    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989147    4918 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989150    4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:27.001658 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989152    4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989155    4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989158    4918 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989160    4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989163    4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989165    4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989168    4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989172    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989175    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989177    4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989180    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989182    4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989185    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989187    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989190    4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989192    4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989195    4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989198    4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989200    4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989203    4918 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:27.002239 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989205    4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989208    4918 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989210    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989213    4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989215    4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989217    4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989220    4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989222    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989226    4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989228    4918 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989231    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989233    4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989236    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989238    4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989241    4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989243    4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989246    4918 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989248    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989251    4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989254    4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:27.002763 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989257    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989259    4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989262    4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989264    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989267    4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989269    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989272    4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989274    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989277    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989279    4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989283    4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989286    4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989289    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989292    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989294    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989297    4918 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989299    4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989302    4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989304    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:27.003245 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989307    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989311    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989314    4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989316    4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989319    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989322    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989324    4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989327    4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989329    4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989332    4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989334    4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989337    4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989340    4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989343    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989345    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989347    4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989350    4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989353    4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989356    4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:27.003730 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989358    4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989361    4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989363    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989366    4918 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989369    4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989372    4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.989374    4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.994324    4918 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.994339    4918 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.994358    4918 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.994363    4918 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994442    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994449    4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994453    4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994456    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994459    4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994462    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994465    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994468    4918 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:27.004204 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994471    4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994473    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994476    4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994479    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994481    4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994484    4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994487    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994490    4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994492    4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994495    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994498    4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994502    4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994506    4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994509    4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994513    4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994516    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994519    4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994521    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994524    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:27.004694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994526    4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994528    4918 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994531    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994534    4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994537    4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994539    4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994541    4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994544    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994547    4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994549    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994551    4918 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994554    4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994556    4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994559    4918 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994561    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994563    4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994566    4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994569    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994571    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994574    4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:27.005144 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994577    4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994579    4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994582    4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994584    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994586    4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994589    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994591    4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994593    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994597    4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994601    4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994604    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994606    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994609    4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994611    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994613    4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994616    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994619    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994622    4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994624    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:27.005706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994627    4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994629    4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994632    4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994634    4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994636    4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994639    4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994641    4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994644    4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994646    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994648    4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994651    4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994653    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994656    4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994659    4918 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994661    4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994664    4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994667    4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994669    4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994672    4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994674    4918 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:27.006169 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994677    4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994679    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994682    4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994684    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994687    4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994690    4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994692    4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994695    4918 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994697    4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994699    4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994703    4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994705    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994708    4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994710    4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994712    4918 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994715    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994718    4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994720    4918 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994723    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:27.006703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994725    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.994754    4918 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994884    4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994889    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994892    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994894    4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994897    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994901    4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994904    4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994906    4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994909    4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994911    4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994914    4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994916    4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994919    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994921    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:27.007211 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994924    4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994926    4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994929    4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994931    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994934    4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994936    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994939    4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994942    4918 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994945    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994947    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994950    4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994952    4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994955    4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994957    4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994959    4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994962    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994964    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994967    4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994969    4918 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994974    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:27.007791 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994976    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994979    4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994981    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994984    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994987    4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994990    4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994992    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994995    4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.994998    4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995000    4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995003    4918 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995005    4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995008    4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995010    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995012    4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995015    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995018    4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995020    4918 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995023    4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995025    4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:27.008451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995028    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995030    4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995032    4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995035    4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995037    4918 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995040    4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995042    4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995045    4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995047    4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995050    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995053    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995057    4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995060    4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995062    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995066    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995068    4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995071    4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995077    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995079    4918 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995082    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:27.009234 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995084    4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995087    4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995091    4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995095    4918 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995098    4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995102    4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995104    4918 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995107    4918 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995109    4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995111    4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995114    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995117    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995119    4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995122    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995124    4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995127    4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995129    4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995131    4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995134    4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:27.010026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995137    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995139    4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995142    4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995144    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995149    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995151    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995154    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995156    4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995159    4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995163    4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995167    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995170    4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995173    4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.995196    4918 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995228    4918 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:01:27.010806 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995232    4918 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995234    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995237    4918 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995240    4918 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995243    4918 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995245    4918 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995248    4918 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995250    4918 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995253    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995255    4918 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995258    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995261    4918 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995263    4918 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995266    4918 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995268    4918 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995271    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995273    4918 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995276    4918 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995278    4918 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995280    4918 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:01:27.011197 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995283    4918 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995285    4918 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995289    4918 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995291    4918 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995294    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995297    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995299    4918 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995301    4918 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995304    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995307    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995310    4918 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995313    4918 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995315    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995318    4918 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995320    4918 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995324    4918 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995326    4918 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995330    4918 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995333    4918 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:01:27.011760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995336    4918 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995339    4918 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995341    4918 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995344    4918 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995346    4918 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995349    4918 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995352    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995354    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995356    4918 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995359    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995361    4918 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995364    4918 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995366    4918 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995369    4918 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995371    4918 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995375    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995377    4918 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995379    4918 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995382    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995384    4918 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:01:27.012312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995404    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995408    4918 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995412    4918 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995417    4918 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995422    4918 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995426    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995429    4918 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995432    4918 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995439    4918 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995441    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995444    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995447    4918 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995449    4918 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995452    4918 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995455    4918 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995457    4918 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995460    4918 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995462    4918 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995465    4918 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995467    4918 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:01:27.013247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995470    4918 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995473    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995475    4918 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995478    4918 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995480    4918 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995483    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995485    4918 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995489    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995491    4918 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995494    4918 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995496    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995499    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995502    4918 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995504    4918 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995507    4918 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995509    4918 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995512    4918 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995515    4918 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995518    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:01:27.014079 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995520    4918 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995524    4918 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995526    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995529    4918 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995532    4918 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995534    4918 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:26.995537    4918 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.995586    4918 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.996533    4918 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.996580    4918 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.996758    4918 server.go:1014] "Starting client certificate rotation"
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.996902    4918 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.996947    4918 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:55:45 +0000 UTC" deadline="2028-04-03 00:31:17.380802057 +0000 UTC"
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.997000    4918 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12483h29m50.383803869s"
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.997204    4918 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.998283    4918 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:01:27.014830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:26.999534    4918 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:01:27.015611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.001123    4918 log.go:25] "Validated CRI v1 image API"
Oct 30 21:01:27.015611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.001882    4918 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:01:27.015611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.003660    4918 fs.go:135] Filesystem UUIDs: map[1381bd88-137e-4352-80bd-e5458225a786:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8662b3f6-4470-4a8e-8c4e-b3f0b1fa4932:/dev/nvme0n1p3]
Oct 30 21:01:27.015787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.003674    4918 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17acf891410fa6976a8a2b36dd876108f6f3e4d53b230a7a5801b5deaf528ab5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17acf891410fa6976a8a2b36dd876108f6f3e4d53b230a7a5801b5deaf528ab5/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1933d1d2a61246b566d9781890abf1d4841a4f89cd7aa43ab6763d5b4f0005ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1933d1d2a61246b566d9781890abf1d4841a4f89cd7aa43ab6763d5b4f0005ac/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ee0ddfc4929dd3b2272af1892c0a9d186bf5a9ec3a383eb652460a2630fda74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ee0ddfc4929dd3b2272af1892c0a9d186bf5a9ec3a383eb652460a2630fda74/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f5cefb7c4ff81375bed60dd0b3aae60eb55bc8a4ba3ccf133b1618e1a0004c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f5cefb7c4ff81375bed60dd0b3aae60eb55bc8a4ba3ccf133b1618e1a0004c4/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/316b7db837bf8eb15989a93a351335526e85adefa4024525139cf16168c90991/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/316b7db837bf8eb15989a93a351335526e85adefa4024525139cf16168c90991/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42898de6490228421f7a8f68d3f5652d8da719ae566852f79ba47fee9f997a8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42898de6490228421f7a8f68d3f5652d8da719ae566852f79ba47fee9f997a8b/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44cc001c54d126f5b3bad814f43e7bef6849925cecb8176809e742a8ae8ab773/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44cc001c54d126f5b3bad814f43e7bef6849925cecb8176809e742a8ae8ab773/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5360740e4869790164337c557c010af80d800091210296279f3c173bdc919b91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5360740e4869790164337c557c010af80d800091210296279f3c173bdc919b91/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61ebb36595e21a7d9e24b5c1f205941d3f025ed75c8ff3b2b0ed16bcfc9b9bb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61ebb36595e21a7d9e24b5c1f205941d3f025ed75c8ff3b2b0ed16bcfc9b9bb1/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/678cfbf8b8db9480d32de5901202ec593769892267f0a058f6f6f843ddd9e7e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/678cfbf8b8db9480d32de5901202ec593769892267f0a058f6f6f843ddd9e7e4/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9af15816b27b2d988ec634dff3db9c8da962fc18f5c0a7986a563729295d27ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9af15816b27b2d988ec634dff3db9c8da962fc18f5c0a7986a563729295d27ab/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1a425e03ff21b01247dddb5563c5e2e576576d0c2ce995f5b8de71f7cd848df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1a425e03ff21b01247dddb5563c5e2e576576d0c2ce995f5b8de71f7cd848df/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3d8e9bfc84976c576ba9dff462d1ef51d4b6042e7dfa9418a04c46100bd55a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3d8e9bfc84976c576ba9dff462d1ef51d4b6042e7dfa9418a04c46100bd55a4/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b34a7ed0210cebfb27a57648633e07b58a6eab52acd0df68b23753d21a94fab3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b34a7ed0210cebfb27a57648633e07b58a6eab52acd0df68b23753d21a94fab3/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8ddbcdd3b7cf9c7d341dab48ff46b3763030d3a8ec0920f8d1c231df8147978/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8ddbcdd3b7cf9c7d341dab48ff46b3763030d3a8ec0920f8d1c231df8147978/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb80dbb42c8818d28f29cbf26f14c62249c49980f775ae61757668f9f8e19c5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb80dbb42c8818d28f29cbf26f14c62249c49980f775ae61757668f9f8e19c5b/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cab1b668c0e5a4af90c1781a2128d9a4ad4e9280f8ca0cd8d2aef24101d1e7c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cab1b668c0e5a4af90c1781a2128d9a4ad4e9280f8ca0cd8d2aef24101d1e7c6/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdf64675c597186b699eb6e8c513a134194d255b16512a48541911865a2d5a74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdf64675c597186b699eb6e8c513a134194d255b16512a48541911865a2d5a74/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf5983c6730cb3c317d46989adcdfd951c86a63a57c8deda9ba3131a24d1bb28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf5983c6730cb3c317d46989adcdfd951c86a63a57c8deda9ba3131a24d1bb28/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5d62d81a997ab8ff4fa9fabcd11777b9f2d3f56bdb052936172689e0b460c31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5d62d81a997ab8ff4fa9fabcd11777b9f2d3f56bdb052936172689e0b460c31/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd8e2e8c7d57d31be34eb25e8e8c3e41bb9ff222a9310d7961bf75b4a9057e9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd8e2e8c7d57d31be34eb25e8e8c3e41bb9ff222a9310d7961bf75b4a9057e9a/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/efb857e7821bb8b237f793852c65f05e1232206434296b808d46c55e506bb33f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/efb857e7821bb8b237f793852c65f05e1232206434296b808d46c55e506bb33f/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5490caf648e02ca6cb15dee373ba2831458ac9635502f34e57224e88deda1c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5490caf648e02ca6cb15dee373ba2831458ac9635502f34e57224e88deda1c4/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/082e6a92-3eef-423d-859e-405adb01c3a1/volumes/kubernetes.io~projected/kube-api-access-wk72h:{mountpoint:/var/lib/kubelet/pods/082e6a92-3eef-423d-859e-405adb01c3a1/volumes/kubernetes.io~projected/kube-api-access-wk72h major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14fc74c3-e4de-4506-a08b-f78dc63fe774/volumes/kubernetes.io~projected/kube-api-access-fnbms:{mountpoint:/var/lib/kubelet/pods/14fc74c3-e4de-4506-a08b-f78dc63fe774/volumes/kubernetes.io~projected/kube-api-access-fnbms major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c305cc1-5455-49d2-a20b-58b81ec38c3a/volumes/kubernetes.io~projected/kube-api-access-hm58p:{mountpoint:/var/lib/kubelet/pods/2c305cc1-5455-49d2-a20b-58b81ec38c3a/volumes/kubernetes.io~projected/kube-api-access-hm58p major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31769be6-0719-4533-997c-e8b458c54d79/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/31769be6-0719-4533-997c-e8b458c54d79/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31769be6-0719-4533-997c-e8b458c54d79/volumes/kubernetes.io~projected/kube-api-access-kjhl2:{mountpoint:/var/lib/kubelet/pods/31769be6-0719-4533-997c-e8b458c54d79/volumes/kubernetes.io~projected/kube-api-access-kjhl2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31769be6-0719-4533-997c-e8b458c54d79/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/31769be6-0719-4533-997c-e8b458c54d79/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a08d3f6-f560-4507-9596-9b548981063c/volumes/kubernetes.io~projected/kube-api-access-tn4gw:{mountpoint:/var/lib/kubelet/pods/3a08d3f6-f560-4507-9596-9b548981063c/volumes/kubernetes.io~projected/kube-api-access-tn4gw major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f89330a-b946-4b7d-8024-5ed5825719e3/volumes/kubernetes.io~projected/kube-api-access-7df44:{mountpoint:/var/lib/kubelet/pods/3f89330a-b946-4b7d-8024-5ed5825719e3/volumes/kubernetes.io~projected/kube-api-access-7df44 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f89330a-b946-4b7d-8024-5ed5825719e3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3f89330a-b946-4b7d-8024-5ed5825719e3/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e6f376c-f439-45e5-9049-d7d097391b83/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4e6f376c-f439-45e5-9049-d7d097391b83/volumes/kubernetes.io~secret/agent-certs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes/kubernetes.io~projected/kube-api-access-k9jdg:{mountpoint:/var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes/kubernetes.io~projected/kube-api-access-k9jdg major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/684225a2-9b4e-4c1f-a784-85da7509bbf1/volumes/kubernetes.io~projected/kube-api-access-fl567:{mountpoint:/var/lib/kubelet/pods/684225a2-9b4e-4c1f-a784-85da7509bbf1/volumes/kubernetes.io~projected/kube-api-access-fl567 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e922433-5427-4340-b27e-cb16e9bd2d0a/volumes/kubernetes.io~projected/kube-api-access-9m7ht:{mountpoint:/var/lib/kubelet/pods/6e922433-5427-4340-b27e-cb16e9bd2d0a/volumes/kubernetes.io~projected/kube-api-access-9m7ht major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/722bd8ce-7062-4e97-a0e8-0984c73c2128/volumes/kubernetes.io~projected/kube-api-access-s8jl8:{mountpoint:/var/lib/kubelet/pods/722bd8ce-7062-4e97-a0e8-0984c73c2128/volumes/kubernetes.io~projected/kube-api-access-s8jl8 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/722bd8ce-7062-4e97-a0e8-0984c73c2128/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/722bd8ce-7062-4e97-a0e8-0984c73c2128/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72689c92-965b-4dfc-9f3c-4b68660f6e53/volumes/kubernetes.io~projected/kube-api-access-xzw62:{mountpoint:/var/lib/kubelet/pods/72689c92-965b-4dfc-9f3c-4b68660f6e53/volumes/kubernetes.io~projected/kube-api-access-xzw62 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72f357cb-3592-424c-9391-9058d0821d5e/volumes/kubernetes.io~projected/kube-api-access-nwd58:{mountpoint:/var/lib/kubelet/pods/72f357cb-3592-424c-9391-9058d0821d5e/volumes/kubernetes.io~projected/kube-api-access-nwd58 major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c27677e-bbe2-4457-b205-1faae8635cb0/volumes/kubernetes.io~projected/kube-api-access-zb264:{mountpoint:/var/lib/kubelet/pods/7c27677e-bbe2-4457-b205-1faae8635cb0/volumes/kubernetes.io~projected/kube-api-access-zb264 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c27677e-bbe2-4457-b205-1faae8635cb0/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/7c27677e-bbe2-4457-b205-1faae8635cb0/volumes/kubernetes.io~secret/default-certificate major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c27677e-bbe2-4457-b205-1faae8635cb0/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/7c27677e-bbe2-4457-b205-1faae8635cb0/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/829cdf21-2065-4e39-902d-7b857a2d74bd/volumes/kubernetes.io~projected/kube-api-access-8jl6z:{mountpoint:/var/lib/kubelet/pods/829cdf21-2065-4e39-902d-7b857a2d74bd/volumes/kubernetes.io~projected/kube-api-access-8jl6z major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86d1bc79-bd2e-422f-b8aa-6de92eabb634/volumes/kubernetes.io~projected/kube-api-access-bj6f4:{mountpoint:/var/lib/kubelet/pods/86d1bc79-bd2e-422f-b8aa-6de92eabb634/volumes/kubernetes.io~projected/kube-api-access-bj6f4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/928df61d-6160-4586-acb9-d13cd90c1a72/volumes/kubernetes.io~projected/kube-api-access-gjgbs:{mountpoint:/var/lib/kubelet/pods/928df61d-6160-4586-acb9-d13cd90c1a72/volumes/kubernetes.io~projected/kube-api-access-gjgbs major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94385c98-9c6d-4fc1-9c48-38c9e5a95447/volumes/kubernetes.io~projected/kube-api-access-p6tln:{mountpoint:/var/lib/kubelet/pods/94385c98-9c6d-4fc1-9c48-38c9e5a95447/volumes/kubernetes.io~projected/kube-api-access-p6tln major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94385c98-9c6d-4fc1-9c48-38c9e5a95447/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/94385c98-9c6d-4fc1-9c48-38c9e5a95447/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98891442-5d51-4e57-a74a-f0fc22de1113/volumes/kubernetes.io~projected/kube-api-access-n6n67:{mountpoint:/var/lib/kubelet/pods/98891442-5d51-4e57-a74a-f0fc22de1113/volumes/kubernetes.io~projected/kube-api-access-n6n67 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5946a39-1450-4d88-aae5-6f76b062fafb/volumes/kubernetes.io~projected/kube-api-access-wr66n:{mountpoint:/var/lib/kubelet/pods/a5946a39-1450-4d88-aae5-6f76b062fafb/volumes/kubernetes.io~projected/kube-api-access-wr66n major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b459c3a5-a5d9-4776-9788-a7d7b943fa53/volumes/kubernetes.io~projected/kube-api-access-l52q4:{mountpoint:/var/lib/kubelet/pods/b459c3a5-a5d9-4776-9788-a7d7b943fa53/volumes/kubernetes.io~projected/kube-api-access-l52q4 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b941fd2b-fbbd-4343-81e1-9a0cdef17a10/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b941fd2b-fbbd-4343-81e1-9a0cdef17a10/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c736b662-1f6a-4984-b82a-9d3b455ee18f/volumes/kubernetes.io~projected/kube-api-access-zjrwb:{mountpoint:/var/lib/kubelet/pods/c736b662-1f6a-4984-b82a-9d3b455ee18f/volumes/kubernetes.io~projected/kube-api-access-zjrwb major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c736b662-1f6a-4984-b82a-9d3b455ee18f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c736b662-1f6a-4984-b82a-9d3b455ee18f/volumes/kubernetes.io~secret/cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes/kubernetes.io~projected/kube-api-access-m9p2b:{mountpoint:/var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes/kubernetes.io~projected/kube-api-access-m9p2b major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d12dc745-07dc-47b3-9658-a13cd5c86638/volumes/kubernetes.io~projected/kube-api-access-r5qw2:{mountpoint:/var/lib/kubelet/pods/d12dc745-07dc-47b3-9658-a13cd5c86638/volumes/kubernetes.io~projected/kube-api-access-r5qw2 major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4a91f73-8795-4a9d-a1cc-55ed355c0312/volumes/kubernetes.io~projected/kube-api-access-4xfpv:{mountpoint:/var/lib/kubelet/pods/d4a91f73-8795-4a9d-a1cc-55ed355c0312/volumes/kubernetes.io~projected/kube-api-access-4xfpv major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4a91f73-8795-4a9d-a1cc-55ed355c0312/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d4a91f73-8795-4a9d-a1cc-55ed355c0312/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d80303a1-1861-4958-a037-891078b48456/volumes/kubernetes.io~projected/kube-api-access-w5jp8:{mountpoint:/var/lib/kubelet/pods/d80303a1-1861-4958-a037-891078b48456/volumes/kubernetes.io~projected/kube-api-access-w5jp8 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dccde73f-6cb5-467f-b650-48195b474db0/volumes/kubernetes.io~projected/kube-api-access-59d6f:{mountpoint:/var/lib/kubelet/pods/dccde73f-6cb5-467f-b650-48195b474db0/volumes/kubernetes.io~projected/kube-api-access-59d6f major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8e758fb-aeed-4b68-ba87-664edc7423ae/volumes/kubernetes.io~projected/kube-api-access-x6n55:{mountpoint:/var/lib/kubelet/pods/e8e758fb-aeed-4b68-ba87-664edc7423ae/volumes/kubernetes.io~projected/kube-api-access-x6n55 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f356dcb1-1e78-4428-bc1f-43ede09e47e3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f356dcb1-1e78-4428-bc1f-43ede09e47e3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f356dcb1-1e78-4428-bc1f-43ede09e47e3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f356dcb1-1e78-4428-bc1f-43ede09e47e3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f356dcb1-1e78-4428-bc1f-43ede09e47e3/volumes/kubernetes.io~projected/kube-api-access-vsq84:{mountpoint:/var/lib/kubelet/pods/f356dcb1-1e78-4428-bc1f-43ede09e47e3/volumes/kubernetes.io~projected/kube-api-access-vsq84 major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/64725d11b94b98fef952bd45e148d19cc4bfa0e228d3e488fc4425b92d131aac/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4f48c10ab1ea072a7b2d79364b52db879105f154e2caaaddbf067a4be52e0eb5/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/153bd8a1265d4a843d911443cccff413eefd3e27dad7dc52c580cca946c44398/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f8bf33151af1514abab5019bfdf9fd339fb787023cd9e7151ec4939dd77edc84/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/fac8605cf79df683102d05c8f19c1f5516f5a681cb6fb303ce8c06c875ebae98/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3442ecf7c52b7483b8b4e4dc9f31aa20e5bd4bf138788b243c45404d4bf92eaf/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/477d7d39149a49fffe59bc7d406c672e80477cb39fb26c5e57653907ecedc958/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/864a3e6cafc315a8f7d1bbfd74bfe2c520b1c7afe8f8522afa4dda6b1358b81f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/9cc29991bf1b14ec0dc2c2ae3744d9ede75ca5b732a79e647d6007c4bbe561b6/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/2d9d011da6299219d3aef9ac170226c1f4af4d446048ff0eb6f173db61e52a89/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/105ec891e4f835eec695e67ce7ce41a908192c82fe48344205c4dc3dd9be3a02/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/89c3bf1aa727098afcee8398d043a635d9e241d10f40d55a0eb618aad890d74e/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d04e9619d7fe68069db9a7b2933d7cb6914654bbf8c99b6237be12e8d8cce524/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d3a79a99c30b143abcf0c94f99b0474c0f86d02598669553efaf0ff23992973e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7123bc63deeb6d33da66d7f6d73cf6be3b49ffb3a83ec84b72d9781eeeebf3c5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/08cc3d9ae4f257ee8c21fa50649bcb0219a6aa326cb8bf77c08f7c1262b9a353/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d2276d6e785d90350c12b139ef8089259e85c3ef1ef7165af51985a15139b193/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/f8efd4dfbe6935db672c1e6af3607526e6ddb8a26fabbefae9c640f63644a0c8/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/45900cce44ea88bdecaafcd879db1e1271922e241544d44f6cabc2b1ae526572/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/04c1ced90b8eb9c22aa8e879810e6a8945e50b4672d8432c7049a84b87941d21/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e863b4db0abe4ed0e9ae01b692bd54e5aeff4be7179aa3e4ff9da92f3f9cdf82/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/eb54d87d6a3b62bec59baac54bbb93a66629d8b99c1c0f7989e601f0fe5287bb/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/52689622b81403b846087216f6830d29b58db699ee9d86475223a598a25c482d/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/06ef3aecbdafd66731ea8850b50fe244edff557c04c95e542e3cd2146c852a69/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/ebe4de15cfdbee61f1317a3c5fc0fce55b7ce8c662cebd5a1b86cd9329955473/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/9d10e6f23180803228831bf55f481427361715d6671f3fd1a9ba233af5ff4e77/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/b608ee12e86418de51a11b305da0a2541bcd7260c0142693e89317eb076d77e3/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/e824ab3895ac86901763f357bd17c10ff37f00451746375421dcf18ace0e609f/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/caa106a751a60e43878017f80a12b240c71fd96f0fc914dff5ee78a555172bb3/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/6920605a5335fa555d9206275303825b3a4c8a2c220df75e826a814a6f7d8ab5/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/5f52a4cc6b4cf36736dd17f0fa3be50ff1fac0041730d7ab861861275c7abcd8/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/0c5032098609b35cb280ddd73ecb27a8fa23468401bd2e079d1d266446be2638/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/f9ffa8de966feaa999a7012aa366db881a61cedfaddec8c2c4e06529ae2374df/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/d4d2af126465133c0d7f69d6200a4bcc31878d77e9eb38e8ed1ed67eb5233838/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/bf59731df8b159ea850c48cfc05ce8f022e0d3a1d73c7bbf112e8c079cbfdf3b/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/a0b1cd193e2ed32880522633ab58cd1f0d317a7453cef5ba1267bd4ae5c7dfe2/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/bea006464464aa2dec6fe5fdffdaf451dadec3b20d9e682594b0e06c3f877d79/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/6eb78b2c2170404929c93941ce408f30316649e5eb9a363aede22c62f524d424/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/d3946333f58a33158c92b6440ad99a3003bba9ed8a204178619df76ca5d1eef5/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/7041a7439178c07d277c405e24a0d93c518191922d721b2153c0ab8117bcbc77/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/ac006c767bed7a4a7e9f03fec17ef4c8d71633ec5342a1f8e9cf8d4b8af2d887/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/3431417bbcc31e294af8a4de23b45adb5ad6c676a3cfd02423cd1b697f831a6b/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/c2e995d1fb7627746a4d3abd29e829a5260397e99f963398658bd7be2935eb7f/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/c83efbf6fcfffa5e1a5fc0268992b26a2950a361265b5311cd6fb5afe3980c1f/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/bdee7e7205ba9257a52262a90d0476838ecda4f36e5370674a5e962d8813b57d/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/3ca678f550b415a17ae3135b6162409893c8bd35d9c7ee9d0dffaf62ed66de8f/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/1b40e1bd9974d3f472e629eb280a52b2226ac4dcfe5b1bd71b36360b0daea6dd/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/5d55bb3b6144f1d71dff5a0a3de60ab0e4b1f032b31d1846fb3a58f946948478/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/44649f8cfb820f64d874e1e08bb6d547c42537ba045add6df75a88afe4192f29/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/04975ad2fb571ee9668e4df023c29cf0866a9d9d0adfa0f25616ee4aaa8a6632/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/0d58ecaaafeafd65530650c6ff89409797a7f76d820fa2eced5e04fece30fad5/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/72abc8fb2a578b58c5f7473747671e23c24d18d1a14066692e52b1bafe9a93f7/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/7760259cf1f99c1a0c10bced38a588e7f4c7e194261c497dc2533727f8f13658/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1bd71fb4280fd2df588b62b80003c349dfbcda538d43df3bf3028e1ad8f3922c/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/9cec508657d4c875f302866d0ac1ba9a35b04bf1bc6f9e9648500d931aaebd24/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/36bb91f0dd1d228f8f4c2f62c8499027ba855360fc3f4c8eda9263b5d2840c8b/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.011698    4918 manager.go:217] Machine: {Timestamp:2026-10-30 21:01:27.011225378 +0000 UTC m=+0.057784516 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100812 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2815ce5eae4c005e02a3f42cd38b47 SystemUUID:ec2815ce-5eae-4c00-5e02-a3f42cd38b47 BootID:384fa65c-8f00-4c73-a693-f91d29a4faa8 Filesystems:[{Device:/var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c736b662-1f6a-4984-b82a-9d3b455ee18f/volumes/kubernetes.io~projected/kube-api-access-zjrwb DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5d62d81a997ab8ff4fa9fabcd11777b9f2d3f56bdb052936172689e0b460c31/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/082e6a92-3eef-423d-859e-405adb01c3a1/volumes/kubernetes.io~projected/kube-api-access-wk72h DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86d1bc79-bd2e-422f-b8aa-6de92eabb634/volumes/kubernetes.io~projected/kube-api-access-bj6f4 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c27677e-bbe2-4457-b205-1faae8635cb0/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/94385c98-9c6d-4fc1-9c48-38c9e5a95447/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f5cefb7c4ff81375bed60dd0b3aae60eb55bc8a4ba3ccf133b1618e1a0004c4/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/72f357cb-3592-424c-9391-9058d0821d5e/volumes/kubernetes.io~projected/kube-api-access-nwd58 DeviceMajor:0 DeviceMinor:326 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e922433-5427-4340-b27e-cb16e9bd2d0a/volumes/kubernetes.io~projected/kube-api-access-9m7ht DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c27677e-bbe2-4457-b205-1faae8635cb0/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61ebb36595e21a7d9e24b5c1f205941d3f025ed75c8ff3b2b0ed16bcfc9b9bb1/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72689c92-965b-4dfc-9f3c-4b68660f6e53/volumes/kubernetes.io~projected/kube-api-access-xzw62 DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e8e758fb-aeed-4b68-ba87-664edc7423ae/volumes/kubernetes.io~projected/kube-api-access-x6n55 DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3f89330a-b946-4b7d-8024-5ed5825719e3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdf64675c597186b699eb6e8c513a134194d255b16512a48541911865a2d5a74/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d12dc745-07dc-47b3-9658-a13cd5c86638/volumes/kubernetes.io~projected/kube-api-access-r5qw2 DeviceMajor:0 DeviceMinor:324 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/14fc74c3-e4de-4506-a08b-f78dc63fe774/volumes/kubernetes.io~projected/kube-api-access-fnbms DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/316b7db837bf8eb15989a93a351335526e85adefa4024525139cf16168c90991/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes/kubernetes.io~projected/kube-api-access-m9p2b DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/31769be6-0719-4533-997c-e8b458c54d79/volumes/kubernetes.io~projected/kube-api-access-kjhl2 DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/722bd8ce-7062-4e97-a0e8-0984c73c2128/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/722bd8ce-7062-4e97-a0e8-0984c73c2128/volumes/kubernetes.io~projected/kube-api-access-s8jl8 DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7c27677e-bbe2-4457-b205-1faae8635cb0/volumes/kubernetes.io~projected/kube-api-access-zb264 DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/94385c98-9c6d-4fc1-9c48-38c9e5a95447/volumes/kubernetes.io~projected/kube-api-access-p6tln DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5360740e4869790164337c557c010af80d800091210296279f3c173bdc919b91/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb80dbb42c8818d28f29cbf26f14c62249c49980f775ae61757668f9f8e19c5b/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a08d3f6-f560-4507-9596-9b548981063c/volumes/kubernetes.io~projected/kube-api-access-tn4gw DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/98891442-5d51-4e57-a74a-f0fc22de1113/volumes/kubernetes.io~projected/kube-api-access-n6n67 DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd8e2e8c7d57d31be34eb25e8e8c3e41bb9ff222a9310d7961bf75b4a9057e9a/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes/kubernetes.io~projected/kube-api-access-k9jdg DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/684225a2-9b4e-4c1f-a784-85da7509bbf1/volumes/kubernetes.io~projected/kube-api-access-fl567 DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b34a7ed0210cebfb27a57648633e07b58a6eab52acd0df68b23753d21a94fab3/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17acf891410fa6976a8a2b36dd876108f6f3e4d53b230a7a5801b5deaf528ab5/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31769be6-0719-4533-997c-e8b458c54d79/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3d8e9bfc84976c576ba9dff462d1ef51d4b6042e7dfa9418a04c46100bd55a4/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/efb857e7821bb8b237f793852c65f05e1232206434296b808d46c55e506bb33f/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cab1b668c0e5a4af90c1781a2128d9a4ad4e9280f8ca0cd8d2aef24101d1e7c6/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/829cdf21-2065-4e39-902d-7b857a2d74bd/volumes/kubernetes.io~projected/kube-api-access-8jl6z DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1a425e03ff21b01247dddb5563c5e2e576576d0c2ce995f5b8de71f7cd848df/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dccde73f-6cb5-467f-b650-48195b474db0/volumes/kubernetes.io~projected/kube-api-access-59d6f DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ee0ddfc4929dd3b2272af1892c0a9d186bf5a9ec3a383eb652460a2630fda74/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3f89330a-b946-4b7d-8024-5ed5825719e3/volumes/kubernetes.io~projected/kube-api-access-7df44 DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/928df61d-6160-4586-acb9-d13cd90c1a72/volumes/kubernetes.io~projected/kube-api-access-gjgbs DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b941fd2b-fbbd-4343-81e1-9a0cdef17a10/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:270 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44cc001c54d126f5b3bad814f43e7bef6849925cecb8176809e742a8ae8ab773/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31769be6-0719-4533-997c-e8b458c54d79/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9af15816b27b2d988ec634dff3db9c8da962fc18f5c0a7986a563729295d27ab/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5490caf648e02ca6cb15dee373ba2831458ac9635502f34e57224e88deda1c4/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f356dcb1-1e78-4428-bc1f-43ede09e47e3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c305cc1-5455-49d2-a20b-58b81ec38c3a/volumes/kubernetes.io~projected/kube-api-access-hm58p DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f356dcb1-1e78-4428-bc1f-43ede09e47e3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b459c3a5-a5d9-4776-9788-a7d7b943fa53/volumes/kubernetes.io~projected/kube-api-access-l52q4 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8ddbcdd3b7cf9c7d341dab48ff46b3763030d3a8ec0920f8d1c231df8147978/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f356dcb1-1e78-4428-bc1f-43ede09e47e3/volumes/kubernetes.io~projected/kube-api-access-vsq84 DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1933d1d2a61246b566d9781890abf1d4841a4f89cd7aa43ab6763d5b4f0005ac/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5946a39-1450-4d88-aae5-6f76b062fafb/volumes/kubernetes.io~projected/kube-api-access-wr66n DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e6f376c-f439-45e5-9049-d7d097391b83/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d4a91f73-8795-4a9d-a1cc-55ed355c0312/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d80303a1-1861-4958-a037-891078b48456/volumes/kubernetes.io~projected/kube-api-access-w5jp8 DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4a91f73-8795-4a9d-a1cc-55ed355c0312/volumes/kubernetes.io~projected/kube-api-access-4xfpv DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42898de6490228421f7a8f68d3f5652d8da719ae566852f79ba47fee9f997a8b/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf5983c6730cb3c317d46989adcdfd951c86a63a57c8deda9ba3131a24d1bb28/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/678cfbf8b8db9480d32de5901202ec593769892267f0a058f6f6f843ddd9e7e4/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c736b662-1f6a-4984-b82a-9d3b455ee18f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:17acf891410fa69 MacAddress:62:0f:ee:e7:21:22 Speed:10000 Mtu:8901} {Name:1ee0ddfc4929dd3 MacAddress:46:cd:50:9d:54:99 Speed:10000 Mtu:8901} {Name:1f5cefb7c4ff813 MacAddress:d2:55:c1:20:9c:b4 Speed:10000 Mtu:8901} {Name:316b7db837bf8eb MacAddress:7a:92:25:10:d4:82 Speed:10000 Mtu:8901} {Name:42898de64902284 MacAddress:f6:5f:04:e4:f1:ba Speed:10000 Mtu:8901} {Name:5360740e4869790 MacAddress:66:31:6e:72:cb:b3 Speed:10000 Mtu:8901} {Name:a3d8e9bfc84976c MacAddress:66:b7:36:23:2d:78 Speed:10000 Mtu:8901} {Name:b34a7ed0210cebf MacAddress:aa:4a:d1:6b:6f:e7 Speed:10000 Mtu:8901} {Name:b8ddbcdd3b7cf9c MacAddress:ae:3d:24:19:e2:3a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e3:89:72:87 Speed:0 Mtu:9001} {Name:br-int MacAddress:ba:3c:59:68:bf:70 Speed:0 Mtu:8901} {Name:cab1b668c0e5a4a MacAddress:8e:75:46:4b:e0:4d Speed:10000 Mtu:8901} {Name:cdf64675c597186 MacAddress:fe:7d:db:c3:92:d2 Speed:10000 Mtu:8901} {Name:cf5983c6730cb3c MacAddress:72:5e:df:13:cb:f8 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e3:89:72:87 Speed:0 Mtu:9001} {Name:f5490caf648e02c MacAddress:4e:5e:c4:4a:cb:34 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:b2:48:0b:05:52:57 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1e:c9:3c:a2:b5:99 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.012016    4918 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.012054    4918 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.012414    4918 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.012448    4918 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-90.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.012640    4918 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.012652    4918 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.012666    4918 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.012690    4918 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.012717    4918 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.012974    4918 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.013110    4918 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.013156    4918 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.013170    4918 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.013184    4918 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.013197    4918 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.013207    4918 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.014105    4918 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.014119    4918 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:01:27.016206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.014589    4918 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:01:27.017704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.017687    4918 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:01:27.017704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.017703    4918 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:01:27.017842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.017830    4918 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:01:27.018056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018043    4918 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:01:27.018129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018083    4918 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:01:27.018205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018194    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:01:27.018262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018214    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:01:27.018262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018224    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:01:27.018262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018235    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:01:27.018262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018243    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:01:27.018262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018251    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:01:27.018262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018260    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:01:27.018723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018269    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:01:27.018723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018279    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:01:27.018723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018288    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:01:27.018723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018300    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:01:27.018723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018315    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:01:27.018723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018349    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:01:27.018723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.018358    4918 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:01:27.022124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.022110    4918 server.go:1263] "Started kubelet"
Oct 30 21:01:27.022205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.022176    4918 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:01:27.022277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.022205    4918 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:01:27.022339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.022325    4918 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:01:27.022634 ip-10-0-139-90 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:01:27.022767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.022651    4918 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:01:27.024338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.024287    4918 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:01:27.027147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.027039    4918 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:01:27.027288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.027131    4918 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.028982    4918 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:55:47 +0000 UTC" deadline="2028-07-24 22:39:52.396477212 +0000 UTC"
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.029021    4918 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15193h38m25.367460003s"
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:27.029152    4918 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-90.us-east-2.compute.internal\" not found"
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.029611    4918 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.029626    4918 factory.go:55] Registering systemd factory
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.029638    4918 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.029709    4918 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.029720    4918 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.029841    4918 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.030190    4918 factory.go:153] Registering CRI-O factory
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.030202    4918 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.030216    4918 factory.go:103] Registering Raw factory
Oct 30 21:01:27.029124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.030225    4918 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:01:27.030981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.030511    4918 manager.go:319] Starting recovery of all containers
Oct 30 21:01:27.032407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032353    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86d1bc79-bd2e-422f-b8aa-6de92eabb634" volumeName="kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:01:27.032407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032383    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94385c98-9c6d-4fc1-9c48-38c9e5a95447" volumeName="kubernetes.io/configmap/94385c98-9c6d-4fc1-9c48-38c9e5a95447-trusted-ca" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032417    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5946a39-1450-4d88-aae5-6f76b062fafb" volumeName="kubernetes.io/configmap/a5946a39-1450-4d88-aae5-6f76b062fafb-telemetry-config" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032430    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f89330a-b946-4b7d-8024-5ed5825719e3" volumeName="kubernetes.io/configmap/3f89330a-b946-4b7d-8024-5ed5825719e3-config" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032442    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f89330a-b946-4b7d-8024-5ed5825719e3" volumeName="kubernetes.io/projected/3f89330a-b946-4b7d-8024-5ed5825719e3-kube-api-access-7df44" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032454    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56aaf442-dd68-4562-bc03-11ba5d35c28e" volumeName="kubernetes.io/empty-dir/56aaf442-dd68-4562-bc03-11ba5d35c28e-ca-trust-extracted" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032464    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86d1bc79-bd2e-422f-b8aa-6de92eabb634" volumeName="kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032476    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94385c98-9c6d-4fc1-9c48-38c9e5a95447" volumeName="kubernetes.io/secret/94385c98-9c6d-4fc1-9c48-38c9e5a95447-serving-cert" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032487    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="082e6a92-3eef-423d-859e-405adb01c3a1" volumeName="kubernetes.io/projected/082e6a92-3eef-423d-859e-405adb01c3a1-kube-api-access-wk72h" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032497    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c27677e-bbe2-4457-b205-1faae8635cb0" volumeName="kubernetes.io/projected/7c27677e-bbe2-4457-b205-1faae8635cb0-kube-api-access-zb264" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032508    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" volumeName="kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-certificates" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032517    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" volumeName="kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-installation-pull-secrets" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032527    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d80303a1-1861-4958-a037-891078b48456" volumeName="kubernetes.io/projected/d80303a1-1861-4958-a037-891078b48456-kube-api-access-w5jp8" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032536    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8e758fb-aeed-4b68-ba87-664edc7423ae" volumeName="kubernetes.io/projected/e8e758fb-aeed-4b68-ba87-664edc7423ae-kube-api-access-x6n55" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032560    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31769be6-0719-4533-997c-e8b458c54d79" volumeName="kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-env-overrides" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032570    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a08d3f6-f560-4507-9596-9b548981063c" volumeName="kubernetes.io/configmap/3a08d3f6-f560-4507-9596-9b548981063c-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:01:27.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032581    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e6f376c-f439-45e5-9049-d7d097391b83" volumeName="kubernetes.io/configmap/4e6f376c-f439-45e5-9049-d7d097391b83-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032592    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="722bd8ce-7062-4e97-a0e8-0984c73c2128" volumeName="kubernetes.io/secret/722bd8ce-7062-4e97-a0e8-0984c73c2128-serving-cert" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032602    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="928df61d-6160-4586-acb9-d13cd90c1a72" volumeName="kubernetes.io/configmap/928df61d-6160-4586-acb9-d13cd90c1a72-config-volume" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032612    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c736b662-1f6a-4984-b82a-9d3b455ee18f" volumeName="kubernetes.io/projected/c736b662-1f6a-4984-b82a-9d3b455ee18f-kube-api-access-zjrwb" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032622    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d12dc745-07dc-47b3-9658-a13cd5c86638" volumeName="kubernetes.io/projected/d12dc745-07dc-47b3-9658-a13cd5c86638-kube-api-access-r5qw2" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032632    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4a91f73-8795-4a9d-a1cc-55ed355c0312" volumeName="kubernetes.io/configmap/d4a91f73-8795-4a9d-a1cc-55ed355c0312-service-ca-bundle" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032643    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c736b662-1f6a-4984-b82a-9d3b455ee18f" volumeName="kubernetes.io/secret/c736b662-1f6a-4984-b82a-9d3b455ee18f-cert" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032655    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4a91f73-8795-4a9d-a1cc-55ed355c0312" volumeName="kubernetes.io/configmap/d4a91f73-8795-4a9d-a1cc-55ed355c0312-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032666    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4a91f73-8795-4a9d-a1cc-55ed355c0312" volumeName="kubernetes.io/projected/d4a91f73-8795-4a9d-a1cc-55ed355c0312-kube-api-access-4xfpv" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032677    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72689c92-965b-4dfc-9f3c-4b68660f6e53" volumeName="kubernetes.io/projected/72689c92-965b-4dfc-9f3c-4b68660f6e53-kube-api-access-xzw62" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032687    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="829cdf21-2065-4e39-902d-7b857a2d74bd" volumeName="kubernetes.io/projected/829cdf21-2065-4e39-902d-7b857a2d74bd-kube-api-access-8jl6z" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032697    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94385c98-9c6d-4fc1-9c48-38c9e5a95447" volumeName="kubernetes.io/configmap/94385c98-9c6d-4fc1-9c48-38c9e5a95447-config" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032715    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b459c3a5-a5d9-4776-9788-a7d7b943fa53" volumeName="kubernetes.io/configmap/b459c3a5-a5d9-4776-9788-a7d7b943fa53-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032727    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b941fd2b-fbbd-4343-81e1-9a0cdef17a10" volumeName="kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032743    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14fc74c3-e4de-4506-a08b-f78dc63fe774" volumeName="kubernetes.io/projected/14fc74c3-e4de-4506-a08b-f78dc63fe774-kube-api-access-fnbms" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032760    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56aaf442-dd68-4562-bc03-11ba5d35c28e" volumeName="kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-kube-api-access-k9jdg" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032772    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="722bd8ce-7062-4e97-a0e8-0984c73c2128" volumeName="kubernetes.io/configmap/722bd8ce-7062-4e97-a0e8-0984c73c2128-config" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032783    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c27677e-bbe2-4457-b205-1faae8635cb0" volumeName="kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-default-certificate" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032794    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86d1bc79-bd2e-422f-b8aa-6de92eabb634" volumeName="kubernetes.io/projected/86d1bc79-bd2e-422f-b8aa-6de92eabb634-kube-api-access-bj6f4" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032804    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" volumeName="kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-trusted-ca" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032815    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" volumeName="kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-kube-api-access-m9p2b" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032827    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4a91f73-8795-4a9d-a1cc-55ed355c0312" volumeName="kubernetes.io/empty-dir/d4a91f73-8795-4a9d-a1cc-55ed355c0312-snapshots" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.032839    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a08d3f6-f560-4507-9596-9b548981063c" volumeName="kubernetes.io/configmap/3a08d3f6-f560-4507-9596-9b548981063c-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033035    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72689c92-965b-4dfc-9f3c-4b68660f6e53" volumeName="kubernetes.io/configmap/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-config" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033050    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b459c3a5-a5d9-4776-9788-a7d7b943fa53" volumeName="kubernetes.io/projected/b459c3a5-a5d9-4776-9788-a7d7b943fa53-kube-api-access-l52q4" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033062    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f356dcb1-1e78-4428-bc1f-43ede09e47e3" volumeName="kubernetes.io/empty-dir/f356dcb1-1e78-4428-bc1f-43ede09e47e3-tmp" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033073    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="082e6a92-3eef-423d-859e-405adb01c3a1" volumeName="kubernetes.io/empty-dir/082e6a92-3eef-423d-859e-405adb01c3a1-tmp-dir" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033083    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="148c1862-34bb-4b75-af99-96c64f2058b8" volumeName="kubernetes.io/configmap/148c1862-34bb-4b75-af99-96c64f2058b8-nginx-conf" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033095    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31769be6-0719-4533-997c-e8b458c54d79" volumeName="kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033105    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e922433-5427-4340-b27e-cb16e9bd2d0a" volumeName="kubernetes.io/projected/6e922433-5427-4340-b27e-cb16e9bd2d0a-kube-api-access-9m7ht" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033118    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94385c98-9c6d-4fc1-9c48-38c9e5a95447" volumeName="kubernetes.io/projected/94385c98-9c6d-4fc1-9c48-38c9e5a95447-kube-api-access-p6tln" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033129    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98891442-5d51-4e57-a74a-f0fc22de1113" volumeName="kubernetes.io/projected/98891442-5d51-4e57-a74a-f0fc22de1113-kube-api-access-n6n67" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033140    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56aaf442-dd68-4562-bc03-11ba5d35c28e" volumeName="kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-bound-sa-token" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033151    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56aaf442-dd68-4562-bc03-11ba5d35c28e" volumeName="kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-installation-pull-secrets" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033161    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86d1bc79-bd2e-422f-b8aa-6de92eabb634" volumeName="kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033172    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="928df61d-6160-4586-acb9-d13cd90c1a72" volumeName="kubernetes.io/projected/928df61d-6160-4586-acb9-d13cd90c1a72-kube-api-access-gjgbs" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033184    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" volumeName="kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-bound-sa-token" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033194    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccde73f-6cb5-467f-b650-48195b474db0" volumeName="kubernetes.io/projected/dccde73f-6cb5-467f-b650-48195b474db0-kube-api-access-59d6f" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033205    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31769be6-0719-4533-997c-e8b458c54d79" volumeName="kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-ovnkube-config" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033215    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a08d3f6-f560-4507-9596-9b548981063c" volumeName="kubernetes.io/projected/3a08d3f6-f560-4507-9596-9b548981063c-kube-api-access-tn4gw" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033236    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="722bd8ce-7062-4e97-a0e8-0984c73c2128" volumeName="kubernetes.io/projected/722bd8ce-7062-4e97-a0e8-0984c73c2128-kube-api-access-s8jl8" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033249    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="829cdf21-2065-4e39-902d-7b857a2d74bd" volumeName="kubernetes.io/configmap/829cdf21-2065-4e39-902d-7b857a2d74bd-serviceca" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033263    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4a91f73-8795-4a9d-a1cc-55ed355c0312" volumeName="kubernetes.io/empty-dir/d4a91f73-8795-4a9d-a1cc-55ed355c0312-tmp" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033276    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c305cc1-5455-49d2-a20b-58b81ec38c3a" volumeName="kubernetes.io/projected/2c305cc1-5455-49d2-a20b-58b81ec38c3a-kube-api-access-hm58p" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033288    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c27677e-bbe2-4457-b205-1faae8635cb0" volumeName="kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-stats-auth" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033301    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5946a39-1450-4d88-aae5-6f76b062fafb" volumeName="kubernetes.io/projected/a5946a39-1450-4d88-aae5-6f76b062fafb-kube-api-access-wr66n" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033314    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f356dcb1-1e78-4428-bc1f-43ede09e47e3" volumeName="kubernetes.io/empty-dir/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-tuned" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033328    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f89330a-b946-4b7d-8024-5ed5825719e3" volumeName="kubernetes.io/secret/3f89330a-b946-4b7d-8024-5ed5825719e3-serving-cert" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033341    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="684225a2-9b4e-4c1f-a784-85da7509bbf1" volumeName="kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033353    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72f357cb-3592-424c-9391-9058d0821d5e" volumeName="kubernetes.io/empty-dir/72f357cb-3592-424c-9391-9058d0821d5e-data-volume" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033366    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72f357cb-3592-424c-9391-9058d0821d5e" volumeName="kubernetes.io/projected/72f357cb-3592-424c-9391-9058d0821d5e-kube-api-access-nwd58" seLinuxMountContext=""
Oct 30 21:01:27.033356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033380    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" volumeName="kubernetes.io/empty-dir/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-ca-trust-extracted" seLinuxMountContext=""
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033412    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31769be6-0719-4533-997c-e8b458c54d79" volumeName="kubernetes.io/projected/31769be6-0719-4533-997c-e8b458c54d79-kube-api-access-kjhl2" seLinuxMountContext=""
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033426    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31769be6-0719-4533-997c-e8b458c54d79" volumeName="kubernetes.io/secret/31769be6-0719-4533-997c-e8b458c54d79-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033439    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56aaf442-dd68-4562-bc03-11ba5d35c28e" volumeName="kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033451    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4a91f73-8795-4a9d-a1cc-55ed355c0312" volumeName="kubernetes.io/secret/d4a91f73-8795-4a9d-a1cc-55ed355c0312-serving-cert" seLinuxMountContext=""
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033464    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56aaf442-dd68-4562-bc03-11ba5d35c28e" volumeName="kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-certificates" seLinuxMountContext=""
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033475    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56aaf442-dd68-4562-bc03-11ba5d35c28e" volumeName="kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-trusted-ca" seLinuxMountContext=""
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033487    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="928df61d-6160-4586-acb9-d13cd90c1a72" volumeName="kubernetes.io/empty-dir/928df61d-6160-4586-acb9-d13cd90c1a72-tmp-dir" seLinuxMountContext=""
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033499    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" volumeName="kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033512    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f356dcb1-1e78-4428-bc1f-43ede09e47e3" volumeName="kubernetes.io/projected/f356dcb1-1e78-4428-bc1f-43ede09e47e3-kube-api-access-vsq84" seLinuxMountContext=""
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033524    4918 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e6f376c-f439-45e5-9049-d7d097391b83" volumeName="kubernetes.io/secret/4e6f376c-f439-45e5-9049-d7d097391b83-agent-certs" seLinuxMountContext=""
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033534    4918 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.033541    4918 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.034635    4918 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.035689    4918 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:01:27.035906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.035897    4918 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:01:27.036826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.036809    4918 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:01:27.050154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.050130    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/1.log"
Oct 30 21:01:27.050320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.050304    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/1.log"
Oct 30 21:01:27.067663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.067606    4918 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:01:27.069032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.069011    4918 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:01:27.069130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.069038    4918 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:01:27.069130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.069066    4918 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:01:27.069130 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:27.069100    4918 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:01:27.073455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.073432    4918 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:01:27.090369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.090333    4918 generic.go:356] "Generic (PLEG): container finished" podID="29fda1c39c64a6789fa2412aae915cee" containerID="7110cbf338332d0dc33d5e135e5562cb83974412cba79f94ec7ce4db02e8f8f2" exitCode=0
Oct 30 21:01:27.094876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.094854    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/1.log"
Oct 30 21:01:27.094998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.094880    4918 generic.go:356] "Generic (PLEG): container finished" podID="94385c98-9c6d-4fc1-9c48-38c9e5a95447" containerID="6208c4ec3241c702493a4d785547fbe681ffe33610f047305caa3bea8d1962b1" exitCode=255
Oct 30 21:01:27.104074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.104052    4918 generic.go:356] "Generic (PLEG): container finished" podID="86d1bc79-bd2e-422f-b8aa-6de92eabb634" containerID="98200d155179fed8f808f637d4568631b021e9d298732f18b2a295867fe71416" exitCode=0
Oct 30 21:01:27.104074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.104074    4918 generic.go:356] "Generic (PLEG): container finished" podID="86d1bc79-bd2e-422f-b8aa-6de92eabb634" containerID="317762c6211a62d02796b66e2b22665a79bdac82f492c517f02776be813d1f36" exitCode=0
Oct 30 21:01:27.104233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.104080    4918 generic.go:356] "Generic (PLEG): container finished" podID="86d1bc79-bd2e-422f-b8aa-6de92eabb634" containerID="b931cd74696edf2560665870d5e0dffe0d25664866df3b90c2d52904d10f1985" exitCode=0
Oct 30 21:01:27.104233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.104087    4918 generic.go:356] "Generic (PLEG): container finished" podID="86d1bc79-bd2e-422f-b8aa-6de92eabb634" containerID="63b4db0ef787507c3f515f3416ef2c3b783fa50f3d73cdd851c9079f9057a7c6" exitCode=0
Oct 30 21:01:27.104233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.104093    4918 generic.go:356] "Generic (PLEG): container finished" podID="86d1bc79-bd2e-422f-b8aa-6de92eabb634" containerID="fd1412fdda0e22902a2491a6122b41eba79a0adc217c00ff16284f7dc0b77514" exitCode=0
Oct 30 21:01:27.104233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.104098    4918 generic.go:356] "Generic (PLEG): container finished" podID="86d1bc79-bd2e-422f-b8aa-6de92eabb634" containerID="f3dff7f82613ac08dfb6881ab39e39ff157d8ed22a744dc354454c5c2d021907" exitCode=0
Oct 30 21:01:27.120728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.120709    4918 manager.go:324] Recovery completed
Oct 30 21:01:27.140635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.140611    4918 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:01:27.140635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.140628    4918 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:01:27.140806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.140645    4918 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:01:27.140806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.140762    4918 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:01:27.140806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.140772    4918 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:01:27.140806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.140788    4918 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:01:27.140806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.140794    4918 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:01:27.140806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.140798    4918 policy_none.go:50] "Start"
Oct 30 21:01:27.140806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.140806    4918 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:01:27.141054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.140814    4918 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:01:27.153816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.153774    4918 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:01:27.153816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.153795    4918 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:01:27.153816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.153801    4918 policy_none.go:44] "Start"
Oct 30 21:01:27.158174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.158149    4918 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:01:27.160673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.160660    4918 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:01:27.161224 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:27.160683    4918 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:01:27.161299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.161228    4918 server.go:87] "Starting device plugin registration server"
Oct 30 21:01:27.161495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.161482    4918 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:01:27.161546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.161497    4918 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:01:27.161632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.161605    4918 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:01:27.161821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.161749    4918 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:01:27.161821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.161817    4918 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:01:27.161821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.161824    4918 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:01:27.161996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.161891    4918 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:01:27.161803657Z","UUID":"b6a3595b-6e45-4651-a451-9e4ce9a384f6","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:01:27.163271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.163241    4918 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:01:27.163358 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:27.163270    4918 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:01:27.163358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.163286    4918 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:01:27.169194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.169158    4918 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"]
Oct 30 21:01:27.169939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.169911    4918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.170032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.169930    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"29fda1c39c64a6789fa2412aae915cee","Type":"ContainerStarted","Data":"581d5ba4cc3146cd3735936cc09bc51b72f841d7264067102629735592de0211"}
Oct 30 21:01:27.170032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.169972    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"29fda1c39c64a6789fa2412aae915cee","Type":"ContainerDied","Data":"7110cbf338332d0dc33d5e135e5562cb83974412cba79f94ec7ce4db02e8f8f2"}
Oct 30 21:01:27.170032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.169991    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"29fda1c39c64a6789fa2412aae915cee","Type":"ContainerStarted","Data":"d5d62d81a997ab8ff4fa9fabcd11777b9f2d3f56bdb052936172689e0b460c31"}
Oct 30 21:01:27.170032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.170029    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"bb77e2dfdbc4a16f3bcede022ac475a7","Type":"ContainerStarted","Data":"d75bf0cbb543c4ff2bbe7a443dd1450b778b5b0717f28bbde7bfd8df08a8d4f1"}
Oct 30 21:01:27.170267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.170048    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal" event={"ID":"bb77e2dfdbc4a16f3bcede022ac475a7","Type":"ContainerStarted","Data":"bb80dbb42c8818d28f29cbf26f14c62249c49980f775ae61757668f9f8e19c5b"}
Oct 30 21:01:27.170267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.170145    4918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.182994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.182977    4918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:01:27.183311 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:27.183283    4918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.183791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.183771    4918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:01:27.183883 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:27.183830    4918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.237453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.237430    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.237566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.237460    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.237566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.237476    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb77e2dfdbc4a16f3bcede022ac475a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"bb77e2dfdbc4a16f3bcede022ac475a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.262474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.262457    4918 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:01:27.264528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.264511    4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:01:27.264619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.264546    4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:01:27.264619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.264561    4918 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-90.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:01:27.264619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.264612    4918 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.291344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.291319    4918 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.291453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.291380    4918 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.337776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.337751    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.337926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.337784    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.337926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.337802    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb77e2dfdbc4a16f3bcede022ac475a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"bb77e2dfdbc4a16f3bcede022ac475a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.337926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.337841    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bb77e2dfdbc4a16f3bcede022ac475a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"bb77e2dfdbc4a16f3bcede022ac475a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.337926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.337865    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:27.337926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:27.337865    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29fda1c39c64a6789fa2412aae915cee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" (UID: \"29fda1c39c64a6789fa2412aae915cee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:28.014050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.014022    4918 apiserver.go:52] "Watching apiserver"
Oct 30 21:01:28.024930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.024902    4918 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:01:28.025585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.025562    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fdn45","openshift-insights/insights-operator-7c4986d449-wg2wk","openshift-insights/insights-runtime-extractor-ltjmm","kube-system/konnectivity-agent-t48nt","openshift-network-console/networking-console-plugin-86c478bb88-22hfw","openshift-ingress/router-default-6d5fbcf987-vhd7f","openshift-cluster-node-tuning-operator/tuned-2z8lm","openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh","openshift-multus/network-metrics-daemon-xppxb","openshift-image-registry/image-registry-67bd499c65-7dz4d","openshift-image-registry/image-registry-67cc99cbd8-gdhf8","openshift-network-operator/iptables-alerter-zbdld","openshift-ingress-canary/ingress-canary-xm2j4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal","openshift-console-operator/console-operator-7cd56b8dbc-22c7r","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c","openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8","openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8","openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc","kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-6slln","openshift-ovn-kubernetes/ovnkube-node-zb8qk","openshift-multus/multus-n8kk5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8","openshift-image-registry/node-ca-rzn4z","openshift-dns/dns-default-hsrbr","openshift-multus/multus-additional-cni-plugins-lwgsc","kube-system/global-pull-secret-syncer-qth9k","kube-system/kas-connection-checker-685895c7c9-m2xqt","openshift-dns/node-resolver-q7r7r","kube-system/kas-connection-checker-685895c7c9-745kc"]
Oct 30 21:01:28.026792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.026774    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:28.027078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.027051    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.027747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.027694    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:28.028939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.028922    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.030004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.029983    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.030169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.030143    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.034238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.031345    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 21:01:28.034238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.032105    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 21:01:28.034238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.032443    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:28.034238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.032451    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:28.034238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.033513    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 21:01:28.034238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.033654    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dcwvc\""
Oct 30 21:01:28.034238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.033835    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5jrbn\""
Oct 30 21:01:28.035510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.035489    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.036045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036026    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.036045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036024    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.036571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036545    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.036685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036665    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-p98j7\""
Oct 30 21:01:28.036770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036559    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 21:01:28.036770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036715    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 21:01:28.036770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036722    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 21:01:28.036770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036641    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.036770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036554    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.036770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036670    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.036770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036715    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hz799\""
Oct 30 21:01:28.037051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036583    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.037051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036590    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 21:01:28.037051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036791    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.037051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036793    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 21:01:28.037051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036806    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 21:01:28.037051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036873    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9rr4h\""
Oct 30 21:01:28.037051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036917    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gdckf\""
Oct 30 21:01:28.037051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036974    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.037051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036982    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:01:28.037051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.036996    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.037051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037047    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.037493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037072    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.037493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037081    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:01:28.037493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037107    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:01:28.037493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037138    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 21:01:28.037493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037153    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.037493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037210    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.037493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037332    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.037493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037347    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 21:01:28.037493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037350    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 21:01:28.037493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037416    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gpwcl\""
Oct 30 21:01:28.037493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037482    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.037968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037950    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.038045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.037944    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 21:01:28.039444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.039429    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 21:01:28.040379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.040361    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 21:01:28.040660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.040646    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:28.040705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.040671    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnbms\" (UniqueName: \"kubernetes.io/projected/14fc74c3-e4de-4506-a08b-f78dc63fe774-kube-api-access-fnbms\") pod \"kas-connection-checker-685895c7c9-m2xqt\" (UID: \"14fc74c3-e4de-4506-a08b-f78dc63fe774\") " pod="kube-system/kas-connection-checker-685895c7c9-m2xqt"
Oct 30 21:01:28.040705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.040691    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a91f73-8795-4a9d-a1cc-55ed355c0312-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:28.040766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.040742    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/722bd8ce-7062-4e97-a0e8-0984c73c2128-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:28.040833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.040810    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm58p\" (UniqueName: \"kubernetes.io/projected/2c305cc1-5455-49d2-a20b-58b81ec38c3a-kube-api-access-hm58p\") pod \"volume-data-source-validator-58c9f5f46d-cwzsc\" (UID: \"2c305cc1-5455-49d2-a20b-58b81ec38c3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc"
Oct 30 21:01:28.040884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.040854    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7ht\" (UniqueName: \"kubernetes.io/projected/6e922433-5427-4340-b27e-cb16e9bd2d0a-kube-api-access-9m7ht\") pod \"network-check-source-6f78d59f99-fvdt8\" (UID: \"6e922433-5427-4340-b27e-cb16e9bd2d0a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8"
Oct 30 21:01:28.040932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.040883    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-image-registry-private-configuration\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.040932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.040910    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-installation-pull-secrets\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.041032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.040934    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-bound-sa-token\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.041032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.040961    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/722bd8ce-7062-4e97-a0e8-0984c73c2128-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:28.041032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.040986    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:28.041032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041009    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7df44\" (UniqueName: \"kubernetes.io/projected/3f89330a-b946-4b7d-8024-5ed5825719e3-kube-api-access-7df44\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:28.041218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041057    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94385c98-9c6d-4fc1-9c48-38c9e5a95447-config\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:28.041218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041104    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6tln\" (UniqueName: \"kubernetes.io/projected/94385c98-9c6d-4fc1-9c48-38c9e5a95447-kube-api-access-p6tln\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:28.041218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041116    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/722bd8ce-7062-4e97-a0e8-0984c73c2128-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:28.041218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041168    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-certificates\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.041218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041195    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-default-certificate\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.041218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041208    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/722bd8ce-7062-4e97-a0e8-0984c73c2128-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:28.041518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041224    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.041518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041254    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4a91f73-8795-4a9d-a1cc-55ed355c0312-serving-cert\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:28.041518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041279    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f89330a-b946-4b7d-8024-5ed5825719e3-config\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:28.041518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041282    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6h7tv\""
Oct 30 21:01:28.041518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041303    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59d6f\" (UniqueName: \"kubernetes.io/projected/dccde73f-6cb5-467f-b650-48195b474db0-kube-api-access-59d6f\") pod \"kas-connection-checker-685895c7c9-745kc\" (UID: \"dccde73f-6cb5-467f-b650-48195b474db0\") " pod="kube-system/kas-connection-checker-685895c7c9-745kc"
Oct 30 21:01:28.041518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041368    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a5946a39-1450-4d88-aae5-6f76b062fafb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:28.041518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041482    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a91f73-8795-4a9d-a1cc-55ed355c0312-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:28.041518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041482    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:28.041904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041572    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/94385c98-9c6d-4fc1-9c48-38c9e5a95447-config\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:28.041904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041621    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 21:01:28.041904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041640    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-certificates\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.041904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041678    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-ca-trust-extracted\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.041904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041812    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-default-certificate\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.041904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041828    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6n55\" (UniqueName: \"kubernetes.io/projected/e8e758fb-aeed-4b68-ba87-664edc7423ae-kube-api-access-x6n55\") pod \"kas-connection-checker-685895c7c9-fdn45\" (UID: \"e8e758fb-aeed-4b68-ba87-664edc7423ae\") " pod="kube-system/kas-connection-checker-685895c7c9-fdn45"
Oct 30 21:01:28.041904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041853    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-ca-trust-extracted\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.041904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041871    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3f89330a-b946-4b7d-8024-5ed5825719e3-config\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:28.042287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041930    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-stats-auth\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.042287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.041929    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d4a91f73-8795-4a9d-a1cc-55ed355c0312-serving-cert\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:28.042287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042075    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d4a91f73-8795-4a9d-a1cc-55ed355c0312-snapshots\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:28.042287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042076    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a5946a39-1450-4d88-aae5-6f76b062fafb-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:28.042287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042132    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94385c98-9c6d-4fc1-9c48-38c9e5a95447-serving-cert\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:28.042287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042175    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.042287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042211    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 21:01:28.042287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042219    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d4a91f73-8795-4a9d-a1cc-55ed355c0312-snapshots\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:28.042287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042265    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hl7k8\""
Oct 30 21:01:28.042287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042260    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-stats-auth\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.042287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042286    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-trusted-ca\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.042768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042296    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/94385c98-9c6d-4fc1-9c48-38c9e5a95447-serving-cert\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:28.042768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042341    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzw62\" (UniqueName: \"kubernetes.io/projected/72689c92-965b-4dfc-9f3c-4b68660f6e53-kube-api-access-xzw62\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:28.042768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042406    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94385c98-9c6d-4fc1-9c48-38c9e5a95447-trusted-ca\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:28.042768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042438    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.042768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042467    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb264\" (UniqueName: \"kubernetes.io/projected/7c27677e-bbe2-4457-b205-1faae8635cb0-kube-api-access-zb264\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.042768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042501    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4a91f73-8795-4a9d-a1cc-55ed355c0312-tmp\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:28.042768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042522    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xfpv\" (UniqueName: \"kubernetes.io/projected/d4a91f73-8795-4a9d-a1cc-55ed355c0312-kube-api-access-4xfpv\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:28.042768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042591    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4a91f73-8795-4a9d-a1cc-55ed355c0312-tmp\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:28.042768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042619    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:28.042768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042653    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f89330a-b946-4b7d-8024-5ed5825719e3-serving-cert\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:28.042768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042688    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr66n\" (UniqueName: \"kubernetes.io/projected/a5946a39-1450-4d88-aae5-6f76b062fafb-kube-api-access-wr66n\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:28.043225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042808    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9p2b\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-kube-api-access-m9p2b\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.043225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042830    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3f89330a-b946-4b7d-8024-5ed5825719e3-serving-cert\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:28.043225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042852    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a91f73-8795-4a9d-a1cc-55ed355c0312-service-ca-bundle\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:28.043225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042880    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8jl8\" (UniqueName: \"kubernetes.io/projected/722bd8ce-7062-4e97-a0e8-0984c73c2128-kube-api-access-s8jl8\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:28.043225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.042998    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4a91f73-8795-4a9d-a1cc-55ed355c0312-service-ca-bundle\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:28.043487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.043268    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:28.043646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.043630    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 21:01:28.043694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.043642    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.045128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.045111    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.046020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.045911    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 21:01:28.046020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.045983    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.046020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.045996    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:01:28.046170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.046022    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 21:01:28.046170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.046024    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 21:01:28.046170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.046095    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.046769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.046757    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.047375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.047361    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.047452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.047420    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.047452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.047430    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.047545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.047530    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.047597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.047527    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.047890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.047866    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-6nsgx\""
Oct 30 21:01:28.047982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.047891    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qd7pl\""
Oct 30 21:01:28.047982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.047970    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mwc48\""
Oct 30 21:01:28.048101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.047995    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qbtrh\""
Oct 30 21:01:28.048101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.048021    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:01:28.048101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.048063    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-99mk5\""
Oct 30 21:01:28.048101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.048080    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-d5xhz\""
Oct 30 21:01:28.048317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.048305    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6m5dr\""
Oct 30 21:01:28.048408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.048366    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 21:01:28.049182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.049167    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:01:28.049217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.049177    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hspc2\""
Oct 30 21:01:28.049217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.049179    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:01:28.049217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.049209    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6bj8t\""
Oct 30 21:01:28.049311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.049220    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:01:28.050804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.049735    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:01:28.050804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.049805    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x8hl8\""
Oct 30 21:01:28.050804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.050096    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:01:28.050804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.050114    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:01:28.050804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.050434    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wx578\""
Oct 30 21:01:28.050804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.050480    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:01:28.050804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.050607    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:01:28.051132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.051028    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:01:28.051132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.051043    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.051132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.051102    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:01:28.051254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.051208    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.051330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.051310    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:01:28.051455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.051441    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.051604    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.051630    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.051734    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.051816    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qfb4q\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.051901    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g68zh\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.051968    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.052160    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v7745\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.052176    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.052303    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cp8br\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.052368    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.052543    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.052842    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.053004    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.053189    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-s7wtw\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.053602    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnbms\" (UniqueName: \"kubernetes.io/projected/14fc74c3-e4de-4506-a08b-f78dc63fe774-kube-api-access-fnbms\") pod \"kas-connection-checker-685895c7c9-m2xqt\" (UID: \"14fc74c3-e4de-4506-a08b-f78dc63fe774\") " pod="kube-system/kas-connection-checker-685895c7c9-m2xqt"
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.053715    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6n55\" (UniqueName: \"kubernetes.io/projected/e8e758fb-aeed-4b68-ba87-664edc7423ae-kube-api-access-x6n55\") pod \"kas-connection-checker-685895c7c9-fdn45\" (UID: \"e8e758fb-aeed-4b68-ba87-664edc7423ae\") " pod="kube-system/kas-connection-checker-685895c7c9-fdn45"
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.053785    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6tln\" (UniqueName: \"kubernetes.io/projected/94385c98-9c6d-4fc1-9c48-38c9e5a95447-kube-api-access-p6tln\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.053966    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.054099    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr66n\" (UniqueName: \"kubernetes.io/projected/a5946a39-1450-4d88-aae5-6f76b062fafb-kube-api-access-wr66n\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.054203    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-bound-sa-token\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.054212    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.054531    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xpqzq\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.054583    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb264\" (UniqueName: \"kubernetes.io/projected/7c27677e-bbe2-4457-b205-1faae8635cb0-kube-api-access-zb264\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.054603    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-lw2cs\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.054697    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m7ht\" (UniqueName: \"kubernetes.io/projected/6e922433-5427-4340-b27e-cb16e9bd2d0a-kube-api-access-9m7ht\") pod \"network-check-source-6f78d59f99-fvdt8\" (UID: \"6e922433-5427-4340-b27e-cb16e9bd2d0a\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fvdt8"
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.054816    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-trusted-ca\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.054829    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.055000    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:01:28.055232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.055075    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-installation-pull-secrets\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.056583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.055289    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59d6f\" (UniqueName: \"kubernetes.io/projected/dccde73f-6cb5-467f-b650-48195b474db0-kube-api-access-59d6f\") pod \"kas-connection-checker-685895c7c9-745kc\" (UID: \"dccde73f-6cb5-467f-b650-48195b474db0\") " pod="kube-system/kas-connection-checker-685895c7c9-745kc"
Oct 30 21:01:28.056583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.055412    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-image-registry-private-configuration\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.056583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.055491    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9p2b\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-kube-api-access-m9p2b\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.056583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.055512    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7df44\" (UniqueName: \"kubernetes.io/projected/3f89330a-b946-4b7d-8024-5ed5825719e3-kube-api-access-7df44\") pod \"service-ca-operator-544d749c8c-2djc8\" (UID: \"3f89330a-b946-4b7d-8024-5ed5825719e3\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8"
Oct 30 21:01:28.056583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.055939    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8jl8\" (UniqueName: \"kubernetes.io/projected/722bd8ce-7062-4e97-a0e8-0984c73c2128-kube-api-access-s8jl8\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c\" (UID: \"722bd8ce-7062-4e97-a0e8-0984c73c2128\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c"
Oct 30 21:01:28.056583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.056128    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm58p\" (UniqueName: \"kubernetes.io/projected/2c305cc1-5455-49d2-a20b-58b81ec38c3a-kube-api-access-hm58p\") pod \"volume-data-source-validator-58c9f5f46d-cwzsc\" (UID: \"2c305cc1-5455-49d2-a20b-58b81ec38c3a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cwzsc"
Oct 30 21:01:28.056583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.056156    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzw62\" (UniqueName: \"kubernetes.io/projected/72689c92-965b-4dfc-9f3c-4b68660f6e53-kube-api-access-xzw62\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:28.056583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.056262    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xfpv\" (UniqueName: \"kubernetes.io/projected/d4a91f73-8795-4a9d-a1cc-55ed355c0312-kube-api-access-4xfpv\") pod \"insights-operator-7c4986d449-wg2wk\" (UID: \"d4a91f73-8795-4a9d-a1cc-55ed355c0312\") " pod="openshift-insights/insights-operator-7c4986d449-wg2wk"
Oct 30 21:01:28.056877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.056739    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94385c98-9c6d-4fc1-9c48-38c9e5a95447-trusted-ca\") pod \"console-operator-7cd56b8dbc-22c7r\" (UID: \"94385c98-9c6d-4fc1-9c48-38c9e5a95447\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:28.121921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.121895    4918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:28.122088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.122071    4918 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:28.131027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.131007    4918 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:01:28.133469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.133449    4918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:01:28.133577 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.133516    4918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:28.134002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.133987    4918 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:01:28.134058 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.134024    4918 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-90.us-east-2.compute.internal"
Oct 30 21:01:28.143852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.143832    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c736b662-1f6a-4984-b82a-9d3b455ee18f-cert\") pod \"ingress-canary-xm2j4\" (UID: \"c736b662-1f6a-4984-b82a-9d3b455ee18f\") " pod="openshift-ingress-canary/ingress-canary-xm2j4"
Oct 30 21:01:28.143942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.143867    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:28.143942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.143897    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-k8s-cni-cncf-io\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.143942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.143926    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.144129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144000    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-kubelet\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.144183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144167    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-slash\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.144236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144200    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qw2\" (UniqueName: \"kubernetes.io/projected/d12dc745-07dc-47b3-9658-a13cd5c86638-kube-api-access-r5qw2\") pod \"migrator-66df5d496b-fw8lh\" (UID: \"d12dc745-07dc-47b3-9658-a13cd5c86638\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh"
Oct 30 21:01:28.144236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144223    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e6f376c-f439-45e5-9049-d7d097391b83-agent-certs\") pod \"konnectivity-agent-t48nt\" (UID: \"4e6f376c-f439-45e5-9049-d7d097391b83\") " pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:01:28.144236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144252    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-original-pull-secret\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:28.144236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144258    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.144236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144252    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c736b662-1f6a-4984-b82a-9d3b455ee18f-cert\") pod \"ingress-canary-xm2j4\" (UID: \"c736b662-1f6a-4984-b82a-9d3b455ee18f\") " pod="openshift-ingress-canary/ingress-canary-xm2j4"
Oct 30 21:01:28.144529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144287    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.144529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144313    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-sys\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.144529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144337    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-tuned\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.144529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144360    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsq84\" (UniqueName: \"kubernetes.io/projected/f356dcb1-1e78-4428-bc1f-43ede09e47e3-kube-api-access-vsq84\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.144529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144432    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.144529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144466    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-bound-sa-token\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144547    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-tuned\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144594    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144616    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e6f376c-f439-45e5-9049-d7d097391b83-agent-certs\") pod \"konnectivity-agent-t48nt\" (UID: \"4e6f376c-f439-45e5-9049-d7d097391b83\") " pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144624    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-ovnkube-config\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144641    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cni-binary-copy\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144656    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-run\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.144638    4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.144694    4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc99cbd8-gdhf8: secret "image-registry-tls" not found
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144700    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-netns\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144736    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.144771    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls podName:ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:28.644751628 +0000 UTC m=+1.691310770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls") pod "image-registry-67cc99cbd8-gdhf8" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7") : secret "image-registry-tls" not found
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144786    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e6a92-3eef-423d-859e-405adb01c3a1-tmp-dir\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144806    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-image-registry-private-configuration\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144830    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e6f376c-f439-45e5-9049-d7d097391b83-konnectivity-ca\") pod \"konnectivity-agent-t48nt\" (UID: \"4e6f376c-f439-45e5-9049-d7d097391b83\") " pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.144845    4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:01:28.144861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144855    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-systemd\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144884    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-certificates\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144892    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-ovnkube-config\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144897    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082e6a92-3eef-423d-859e-405adb01c3a1-tmp-dir\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.144913    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls podName:72689c92-965b-4dfc-9f3c-4b68660f6e53 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:28.644894357 +0000 UTC m=+1.691453499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fxrz8" (UID: "72689c92-965b-4dfc-9f3c-4b68660f6e53") : secret "samples-operator-tls" not found
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.144934    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-registration-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145036    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/829cdf21-2065-4e39-902d-7b857a2d74bd-serviceca\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145061    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-run-netns\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145060    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-certificates\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145086    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e6f376c-f439-45e5-9049-d7d097391b83-konnectivity-ca\") pod \"konnectivity-agent-t48nt\" (UID: \"4e6f376c-f439-45e5-9049-d7d097391b83\") " pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145094    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-var-lib-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145117    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-cni-netd\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145150    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145171    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-image-registry-private-configuration\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145178    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-etc-selinux\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145243    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/829cdf21-2065-4e39-902d-7b857a2d74bd-serviceca\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145284    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6n67\" (UniqueName: \"kubernetes.io/projected/98891442-5d51-4e57-a74a-f0fc22de1113-kube-api-access-n6n67\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145309    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l52q4\" (UniqueName: \"kubernetes.io/projected/b459c3a5-a5d9-4776-9788-a7d7b943fa53-kube-api-access-l52q4\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145326    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-ovn\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145366    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-hostroot\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145451    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5jp8\" (UniqueName: \"kubernetes.io/projected/d80303a1-1861-4958-a037-891078b48456-kube-api-access-w5jp8\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:28.145519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145483    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk72h\" (UniqueName: \"kubernetes.io/projected/082e6a92-3eef-423d-859e-405adb01c3a1-kube-api-access-wk72h\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145562    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72f357cb-3592-424c-9391-9058d0821d5e-data-volume\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145665    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928df61d-6160-4586-acb9-d13cd90c1a72-tmp-dir\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145739    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72f357cb-3592-424c-9391-9058d0821d5e-data-volume\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145756    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145797    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/928df61d-6160-4586-acb9-d13cd90c1a72-tmp-dir\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145801    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysconfig\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145853    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b459c3a5-a5d9-4776-9788-a7d7b943fa53-host-slash\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145877    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl567\" (UniqueName: \"kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567\") pod \"network-check-target-6slln\" (UID: \"684225a2-9b4e-4c1f-a784-85da7509bbf1\") " pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145891    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-device-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145939    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145978    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-log-socket\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.145997    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-cni-bin\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146061    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-kubelet\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146090    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-conf-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146141    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146146    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146172    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72f357cb-3592-424c-9391-9058d0821d5e-crio-socket\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146192    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysctl-d\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.146203    4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146209    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/928df61d-6160-4586-acb9-d13cd90c1a72-config-volume\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.146271    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:28.646254269 +0000 UTC m=+1.692813408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : secret "router-metrics-certs-default" not found
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146308    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjrwb\" (UniqueName: \"kubernetes.io/projected/c736b662-1f6a-4984-b82a-9d3b455ee18f-kube-api-access-zjrwb\") pod \"ingress-canary-xm2j4\" (UID: \"c736b662-1f6a-4984-b82a-9d3b455ee18f\") " pod="openshift-ingress-canary/ingress-canary-xm2j4"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146341    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-system-cni-dir\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.146384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146374    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/928df61d-6160-4586-acb9-d13cd90c1a72-config-volume\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146419    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-run-ovn-kubernetes\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146455    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146507    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56aaf442-dd68-4562-bc03-11ba5d35c28e-ca-trust-extracted\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146533    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-installation-pull-secrets\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146562    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146582    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysctl-conf\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146606    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56aaf442-dd68-4562-bc03-11ba5d35c28e-ca-trust-extracted\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146613    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-kubernetes\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146639    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-lib-modules\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146736    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f356dcb1-1e78-4428-bc1f-43ede09e47e3-tmp\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146765    4918 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146789    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-installation-pull-secrets\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146800    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-system-cni-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146827    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-cni-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146832    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f356dcb1-1e78-4428-bc1f-43ede09e47e3-tmp\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146849    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-etc-kubernetes\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146886    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-modprobe-d\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146901    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146915    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-cni-bin\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146937    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-sys-fs\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146957    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b459c3a5-a5d9-4776-9788-a7d7b943fa53-iptables-alerter-script\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.146992    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-ovnkube-script-lib\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147047    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147063    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-cnibin\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147085    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-trusted-ca\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147108    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-systemd-units\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147136    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jdg\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-kube-api-access-k9jdg\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147164    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147180    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/829cdf21-2065-4e39-902d-7b857a2d74bd-host\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147194    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-dbus\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147211    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-cni-multus\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147212    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b459c3a5-a5d9-4776-9788-a7d7b943fa53-iptables-alerter-script\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147228    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4gw\" (UniqueName: \"kubernetes.io/projected/3a08d3f6-f560-4507-9596-9b548981063c-kube-api-access-tn4gw\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147246    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jl6z\" (UniqueName: \"kubernetes.io/projected/829cdf21-2065-4e39-902d-7b857a2d74bd-kube-api-access-8jl6z\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147264    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-multus-certs\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147359    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31769be6-0719-4533-997c-e8b458c54d79-ovn-node-metrics-cert\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.147370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147360    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-trusted-ca\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147422    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhl2\" (UniqueName: \"kubernetes.io/projected/31769be6-0719-4533-997c-e8b458c54d79-kube-api-access-kjhl2\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147566    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31769be6-0719-4533-997c-e8b458c54d79-ovn-node-metrics-cert\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147612    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-ovnkube-script-lib\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147778    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/082e6a92-3eef-423d-859e-405adb01c3a1-hosts-file\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147811    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-socket-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147841    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj6f4\" (UniqueName: \"kubernetes.io/projected/86d1bc79-bd2e-422f-b8aa-6de92eabb634-kube-api-access-bj6f4\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147868    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147895    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cnibin\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147928    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwd58\" (UniqueName: \"kubernetes.io/projected/72f357cb-3592-424c-9391-9058d0821d5e-kube-api-access-nwd58\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147955    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-node-log\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.147978    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-host\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148013    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-kubelet-config\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148039    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a08d3f6-f560-4507-9596-9b548981063c-multus-daemon-config\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148071    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.148098    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:28.648085373 +0000 UTC m=+1.694644512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : configmap references non-existent config key: service-ca.crt
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148123    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a08d3f6-f560-4507-9596-9b548981063c-cni-binary-copy\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148145    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148165    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/148c1862-34bb-4b75-af99-96c64f2058b8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148181    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-etc-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148197    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-systemd\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148210    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-var-lib-kubelet\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148226    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjgbs\" (UniqueName: \"kubernetes.io/projected/928df61d-6160-4586-acb9-d13cd90c1a72-kube-api-access-gjgbs\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148244    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148285    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-env-overrides\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.148580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148300    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-socket-dir-parent\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.149274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148599    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31769be6-0719-4533-997c-e8b458c54d79-env-overrides\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.149274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148602    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a08d3f6-f560-4507-9596-9b548981063c-multus-daemon-config\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.149274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148630    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/148c1862-34bb-4b75-af99-96c64f2058b8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:28.149274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.148653    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a08d3f6-f560-4507-9596-9b548981063c-cni-binary-copy\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.149831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.149810    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a5946a39-1450-4d88-aae5-6f76b062fafb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-shzqw\" (UID: \"a5946a39-1450-4d88-aae5-6f76b062fafb\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:28.154781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.154761    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-bound-sa-token\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.156266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.156246    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk72h\" (UniqueName: \"kubernetes.io/projected/082e6a92-3eef-423d-859e-405adb01c3a1-kube-api-access-wk72h\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:01:28.156538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.156514    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwd58\" (UniqueName: \"kubernetes.io/projected/72f357cb-3592-424c-9391-9058d0821d5e-kube-api-access-nwd58\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.157358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.156998    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsq84\" (UniqueName: \"kubernetes.io/projected/f356dcb1-1e78-4428-bc1f-43ede09e47e3-kube-api-access-vsq84\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.157358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.157014    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5jp8\" (UniqueName: \"kubernetes.io/projected/d80303a1-1861-4958-a037-891078b48456-kube-api-access-w5jp8\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:28.157493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.157436    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjgbs\" (UniqueName: \"kubernetes.io/projected/928df61d-6160-4586-acb9-d13cd90c1a72-kube-api-access-gjgbs\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:28.157493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.157444    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjrwb\" (UniqueName: \"kubernetes.io/projected/c736b662-1f6a-4984-b82a-9d3b455ee18f-kube-api-access-zjrwb\") pod \"ingress-canary-xm2j4\" (UID: \"c736b662-1f6a-4984-b82a-9d3b455ee18f\") " pod="openshift-ingress-canary/ingress-canary-xm2j4"
Oct 30 21:01:28.157493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.157464    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jl6z\" (UniqueName: \"kubernetes.io/projected/829cdf21-2065-4e39-902d-7b857a2d74bd-kube-api-access-8jl6z\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:01:28.157493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.157465    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4gw\" (UniqueName: \"kubernetes.io/projected/3a08d3f6-f560-4507-9596-9b548981063c-kube-api-access-tn4gw\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.157708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.157464    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qw2\" (UniqueName: \"kubernetes.io/projected/d12dc745-07dc-47b3-9658-a13cd5c86638-kube-api-access-r5qw2\") pod \"migrator-66df5d496b-fw8lh\" (UID: \"d12dc745-07dc-47b3-9658-a13cd5c86638\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fw8lh"
Oct 30 21:01:28.157708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.157565    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6n67\" (UniqueName: \"kubernetes.io/projected/98891442-5d51-4e57-a74a-f0fc22de1113-kube-api-access-n6n67\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.157917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.157890    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj6f4\" (UniqueName: \"kubernetes.io/projected/86d1bc79-bd2e-422f-b8aa-6de92eabb634-kube-api-access-bj6f4\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.158129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.158112    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l52q4\" (UniqueName: \"kubernetes.io/projected/b459c3a5-a5d9-4776-9788-a7d7b943fa53-kube-api-access-l52q4\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:01:28.158129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.158126    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jdg\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-kube-api-access-k9jdg\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.158296 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.158281    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjhl2\" (UniqueName: \"kubernetes.io/projected/31769be6-0719-4533-997c-e8b458c54d79-kube-api-access-kjhl2\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.158712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.158693    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl567\" (UniqueName: \"kubernetes.io/projected/684225a2-9b4e-4c1f-a784-85da7509bbf1-kube-api-access-fl567\") pod \"network-check-target-6slln\" (UID: \"684225a2-9b4e-4c1f-a784-85da7509bbf1\") " pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:28.249379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249349    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-kubelet\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.249533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249408    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-conf-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.249533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249445    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72f357cb-3592-424c-9391-9058d0821d5e-crio-socket\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.249533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249470    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysctl-d\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.249533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249473    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-kubelet\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.249533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249495    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-system-cni-dir\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.249533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249525    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-run-ovn-kubernetes\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249532    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-conf-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249545    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72f357cb-3592-424c-9391-9058d0821d5e-crio-socket\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249551    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysctl-conf\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249585    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-kubernetes\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249603    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-lib-modules\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249611    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-run-ovn-kubernetes\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249583    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-system-cni-dir\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249612    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysctl-d\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249654    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysctl-conf\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249658    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-system-cni-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249663    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-kubernetes\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249681    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-cni-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249701    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-etc-kubernetes\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249702    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-lib-modules\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249717    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-system-cni-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249745    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-etc-kubernetes\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249762    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-modprobe-d\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249776    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-cni-dir\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.249815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249805    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-cni-bin\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249832    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-sys-fs\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249857    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249888    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-modprobe-d\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249905    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-sys-fs\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249894    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-cnibin\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249945    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-cnibin\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249947    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-systemd-units\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249866    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-cni-bin\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249987    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.249986    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-systemd-units\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250016    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/829cdf21-2065-4e39-902d-7b857a2d74bd-host\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250043    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-dbus\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250067    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-cni-multus\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.250082    4918 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250086    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/829cdf21-2065-4e39-902d-7b857a2d74bd-host\") pod \"node-ca-rzn4z\" (UID: \"829cdf21-2065-4e39-902d-7b857a2d74bd\") " pod="openshift-image-registry/node-ca-rzn4z"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250095    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-multus-certs\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250100    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-dbus\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.250133    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert podName:148c1862-34bb-4b75-af99-96c64f2058b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:28.750120444 +0000 UTC m=+1.796679567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-22hfw" (UID: "148c1862-34bb-4b75-af99-96c64f2058b8") : secret "networking-console-plugin-cert" not found
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250135    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-var-lib-cni-multus\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250147    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/082e6a92-3eef-423d-859e-405adb01c3a1-hosts-file\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250169    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-socket-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250147    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-multus-certs\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250191    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/082e6a92-3eef-423d-859e-405adb01c3a1-hosts-file\") pod \"node-resolver-q7r7r\" (UID: \"082e6a92-3eef-423d-859e-405adb01c3a1\") " pod="openshift-dns/node-resolver-q7r7r"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250196    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cnibin\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250224    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-node-log\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250229    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-cnibin\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250247    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-host\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250266    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-node-log\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250274    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-kubelet-config\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250260    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-socket-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250316    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-host\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250323    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b941fd2b-fbbd-4343-81e1-9a0cdef17a10-kubelet-config\") pod \"global-pull-secret-syncer-qth9k\" (UID: \"b941fd2b-fbbd-4343-81e1-9a0cdef17a10\") " pod="kube-system/global-pull-secret-syncer-qth9k"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250352    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250369    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250407    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-etc-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250423    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86d1bc79-bd2e-422f-b8aa-6de92eabb634-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lwgsc\" (UID: \"86d1bc79-bd2e-422f-b8aa-6de92eabb634\") " pod="openshift-multus/multus-additional-cni-plugins-lwgsc"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250428    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-systemd\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250445    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-var-lib-kubelet\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250454    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-etc-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250464    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250482    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-systemd\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250490    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-socket-dir-parent\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.250501    4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.250516    4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd499c65-7dz4d: secret "image-registry-tls" not found
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250516    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-k8s-cni-cncf-io\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250524    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-var-lib-kubelet\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250540    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250564    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-k8s-cni-cncf-io\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250543    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250595    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250602    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-kubelet\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250622    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-multus-socket-dir-parent\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250630    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-slash\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250645    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-kubelet\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250659    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.250680    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls podName:56aaf442-dd68-4562-bc03-11ba5d35c28e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:28.750664352 +0000 UTC m=+1.797223483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls") pod "image-registry-67bd499c65-7dz4d" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e") : secret "image-registry-tls" not found
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250683    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-slash\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.250662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250709    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-sys\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250767    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250777    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-sys\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250796    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-run\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250851    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-run\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.250866    4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.250923    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:28.750901461 +0000 UTC m=+1.797460597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.250943    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-netns\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251019    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-systemd\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251040    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-host-run-netns\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251058    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-registration-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251067    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72f357cb-3592-424c-9391-9058d0821d5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251086    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-run-netns\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251089    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-systemd\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251109    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-var-lib-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251117    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-registration-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251128    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-run-netns\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251146    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-var-lib-openvswitch\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251156    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-cni-netd\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251184    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251206    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-etc-selinux\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251229    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-cni-netd\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251259    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-etc-selinux\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.251267    4918 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251284    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-ovn\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251312    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-hostroot\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251313    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-run-ovn\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.251326    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs podName:d80303a1-1861-4958-a037-891078b48456 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:28.751310949 +0000 UTC m=+1.797870086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs") pod "network-metrics-daemon-xppxb" (UID: "d80303a1-1861-4958-a037-891078b48456") : secret "metrics-daemon-secret" not found
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251363    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251383    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a08d3f6-f560-4507-9596-9b548981063c-hostroot\") pod \"multus-n8kk5\" (UID: \"3a08d3f6-f560-4507-9596-9b548981063c\") " pod="openshift-multus/multus-n8kk5"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251421    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysconfig\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251445    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b459c3a5-a5d9-4776-9788-a7d7b943fa53-host-slash\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251457    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251459    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-device-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251485    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-log-socket\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251508    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f356dcb1-1e78-4428-bc1f-43ede09e47e3-etc-sysconfig\") pod \"tuned-2z8lm\" (UID: \"f356dcb1-1e78-4428-bc1f-43ede09e47e3\") " pod="openshift-cluster-node-tuning-operator/tuned-2z8lm"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251514    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b459c3a5-a5d9-4776-9788-a7d7b943fa53-host-slash\") pod \"iptables-alerter-zbdld\" (UID: \"b459c3a5-a5d9-4776-9788-a7d7b943fa53\") " pod="openshift-network-operator/iptables-alerter-zbdld"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251522    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-log-socket\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251539    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98891442-5d51-4e57-a74a-f0fc22de1113-device-dir\") pod \"aws-ebs-csi-driver-node-dt4vp\" (UID: \"98891442-5d51-4e57-a74a-f0fc22de1113\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dt4vp"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251572    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-cni-bin\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.251661    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31769be6-0719-4533-997c-e8b458c54d79-host-cni-bin\") pod \"ovnkube-node-zb8qk\" (UID: \"31769be6-0719-4533-997c-e8b458c54d79\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:28.252641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.252454    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/928df61d-6160-4586-acb9-d13cd90c1a72-metrics-tls\") pod \"dns-default-hsrbr\" (UID: \"928df61d-6160-4586-acb9-d13cd90c1a72\") " pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:28.327990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.327930    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"
Oct 30 21:01:28.328591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.328572    4918 scope.go:122] "RemoveContainer" containerID="6208c4ec3241c702493a4d785547fbe681ffe33610f047305caa3bea8d1962b1"
Oct 30 21:01:28.343954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.343933    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:28.405187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.404355    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q7r7r_082e6a92-3eef-423d-859e-405adb01c3a1/dns-node-resolver/0.log"
Oct 30 21:01:28.625189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.625161    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw"]
Oct 30 21:01:28.627575 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:28.627550    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5946a39_1450_4d88_aae5_6f76b062fafb.slice/crio-3210dcf22066175b0de768274eb7bb6c4522c53dd7262f4181eef83f6c1e2c8d WatchSource:0}: Error finding container 3210dcf22066175b0de768274eb7bb6c4522c53dd7262f4181eef83f6c1e2c8d: Status 404 returned error can't find the container with id 3210dcf22066175b0de768274eb7bb6c4522c53dd7262f4181eef83f6c1e2c8d
Oct 30 21:01:28.629595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.629580    4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:01:28.655895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.655871    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.656003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.655925    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:28.656003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.655958    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:28.656066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.656001    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:28.656066 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.656030    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:29.656009984 +0000 UTC m=+2.702569120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : configmap references non-existent config key: service-ca.crt
Oct 30 21:01:28.656178 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.656083    4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:01:28.656178 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.656091    4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:28.656178 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.656108    4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc99cbd8-gdhf8: secret "image-registry-tls" not found
Oct 30 21:01:28.656178 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.656127    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls podName:72689c92-965b-4dfc-9f3c-4b68660f6e53 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:29.65611603 +0000 UTC m=+2.702675157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fxrz8" (UID: "72689c92-965b-4dfc-9f3c-4b68660f6e53") : secret "samples-operator-tls" not found
Oct 30 21:01:28.656178 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.656083    4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:01:28.656178 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.656159    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls podName:ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:29.656145094 +0000 UTC m=+2.702704217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls") pod "image-registry-67cc99cbd8-gdhf8" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7") : secret "image-registry-tls" not found
Oct 30 21:01:28.656487 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.656188    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:29.656167412 +0000 UTC m=+2.702726536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : secret "router-metrics-certs-default" not found
Oct 30 21:01:28.658561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.658535    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hsrbr"]
Oct 30 21:01:28.661408 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:28.661372    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928df61d_6160_4586_acb9_d13cd90c1a72.slice/crio-589558961f41909439e274a4fce2135a5dde11d241ff754b85dc8a4d1e7e5824 WatchSource:0}: Error finding container 589558961f41909439e274a4fce2135a5dde11d241ff754b85dc8a4d1e7e5824: Status 404 returned error can't find the container with id 589558961f41909439e274a4fce2135a5dde11d241ff754b85dc8a4d1e7e5824
Oct 30 21:01:28.757278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.757253    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:28.757433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.757305    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:28.757433 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.757409    4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:28.757433 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.757418    4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:28.757433 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.757432    4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd499c65-7dz4d: secret "image-registry-tls" not found
Oct 30 21:01:28.757557 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.757467    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:29.757437665 +0000 UTC m=+2.803996801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:28.757557 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.757480    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls podName:56aaf442-dd68-4562-bc03-11ba5d35c28e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:29.75747364 +0000 UTC m=+2.804032763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls") pod "image-registry-67bd499c65-7dz4d" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e") : secret "image-registry-tls" not found
Oct 30 21:01:28.757557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.757490    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:28.757557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:28.757545    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:28.757712 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.757615    4918 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:01:28.757712 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.757641    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert podName:148c1862-34bb-4b75-af99-96c64f2058b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:29.757635086 +0000 UTC m=+2.804194209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-22hfw" (UID: "148c1862-34bb-4b75-af99-96c64f2058b8") : secret "networking-console-plugin-cert" not found
Oct 30 21:01:28.757712 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.757669    4918 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:01:28.757819 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:28.757716    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs podName:d80303a1-1861-4958-a037-891078b48456 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:29.757699934 +0000 UTC m=+2.804259080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs") pod "network-metrics-daemon-xppxb" (UID: "d80303a1-1861-4958-a037-891078b48456") : secret "metrics-daemon-secret" not found
Oct 30 21:01:29.130668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.130637    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hsrbr" event={"ID":"928df61d-6160-4586-acb9-d13cd90c1a72","Type":"ContainerStarted","Data":"589558961f41909439e274a4fce2135a5dde11d241ff754b85dc8a4d1e7e5824"}
Oct 30 21:01:29.132107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.132086    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw" event={"ID":"a5946a39-1450-4d88-aae5-6f76b062fafb","Type":"ContainerStarted","Data":"3210dcf22066175b0de768274eb7bb6c4522c53dd7262f4181eef83f6c1e2c8d"}
Oct 30 21:01:29.133515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.133488    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:01:29.133987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.133971    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/1.log"
Oct 30 21:01:29.134121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.134003    4918 generic.go:356] "Generic (PLEG): container finished" podID="94385c98-9c6d-4fc1-9c48-38c9e5a95447" containerID="ebe9de9c294b9d969f5d863ce15320b42c635db3b7988e65e196c4ef7db4db65" exitCode=255
Oct 30 21:01:29.134121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.134039    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r" event={"ID":"94385c98-9c6d-4fc1-9c48-38c9e5a95447","Type":"ContainerDied","Data":"ebe9de9c294b9d969f5d863ce15320b42c635db3b7988e65e196c4ef7db4db65"}
Oct 30 21:01:29.134121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.134086    4918 scope.go:122] "RemoveContainer" containerID="6208c4ec3241c702493a4d785547fbe681ffe33610f047305caa3bea8d1962b1"
Oct 30 21:01:29.134663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.134643    4918 scope.go:122] "RemoveContainer" containerID="ebe9de9c294b9d969f5d863ce15320b42c635db3b7988e65e196c4ef7db4db65"
Oct 30 21:01:29.134821 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.134798    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-22c7r_openshift-console-operator(94385c98-9c6d-4fc1-9c48-38c9e5a95447)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r" podUID="94385c98-9c6d-4fc1-9c48-38c9e5a95447"
Oct 30 21:01:29.401085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.401005    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rzn4z_829cdf21-2065-4e39-902d-7b857a2d74bd/node-ca/0.log"
Oct 30 21:01:29.665972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.665896    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:29.665972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.665945    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:29.666189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.665985    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:29.666189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.666030    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:29.666189 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.666157    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.666139694 +0000 UTC m=+4.712698837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : configmap references non-existent config key: service-ca.crt
Oct 30 21:01:29.666648 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.666626    4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:29.666648 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.666646    4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc99cbd8-gdhf8: secret "image-registry-tls" not found
Oct 30 21:01:29.666756 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.666694    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls podName:ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.666678797 +0000 UTC m=+4.713237925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls") pod "image-registry-67cc99cbd8-gdhf8" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7") : secret "image-registry-tls" not found
Oct 30 21:01:29.666791 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.666770    4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:01:29.666823 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.666802    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls podName:72689c92-965b-4dfc-9f3c-4b68660f6e53 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.666790951 +0000 UTC m=+4.713350090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fxrz8" (UID: "72689c92-965b-4dfc-9f3c-4b68660f6e53") : secret "samples-operator-tls" not found
Oct 30 21:01:29.666856 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.666850    4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:01:29.666893 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.666878    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.666867774 +0000 UTC m=+4.713426913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : secret "router-metrics-certs-default" not found
Oct 30 21:01:29.766837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.766799    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:29.767014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.766886    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:29.767014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.766937    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:29.767014 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.766958    4918 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:01:29.767014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:29.766972    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:29.767014 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.767017    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert podName:148c1862-34bb-4b75-af99-96c64f2058b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.76700264 +0000 UTC m=+4.813561790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-22hfw" (UID: "148c1862-34bb-4b75-af99-96c64f2058b8") : secret "networking-console-plugin-cert" not found
Oct 30 21:01:29.767279 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.767056    4918 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:01:29.767279 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.767102    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs podName:d80303a1-1861-4958-a037-891078b48456 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.767091305 +0000 UTC m=+4.813650427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs") pod "network-metrics-daemon-xppxb" (UID: "d80303a1-1861-4958-a037-891078b48456") : secret "metrics-daemon-secret" not found
Oct 30 21:01:29.767279 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.767144    4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:29.767279 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.767151    4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd499c65-7dz4d: secret "image-registry-tls" not found
Oct 30 21:01:29.767279 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.767171    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls podName:56aaf442-dd68-4562-bc03-11ba5d35c28e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.767164809 +0000 UTC m=+4.813723931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls") pod "image-registry-67bd499c65-7dz4d" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e") : secret "image-registry-tls" not found
Oct 30 21:01:29.767279 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.767204    4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:29.767279 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:29.767224    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:31.767217994 +0000 UTC m=+4.813777116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:30.140662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:30.140622    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:01:30.200681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:30.200649    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xm2j4_c736b662-1f6a-4984-b82a-9d3b455ee18f/serve-healthcheck-canary/0.log"
Oct 30 21:01:30.401896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:30.401872    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fw8lh_d12dc745-07dc-47b3-9658-a13cd5c86638/migrator/0.log"
Oct 30 21:01:30.805885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:30.805856    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c_722bd8ce-7062-4e97-a0e8-0984c73c2128/kube-storage-version-migrator-operator/0.log"
Oct 30 21:01:30.991488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:30.991457    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:01:31.145552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:31.145454    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hsrbr" event={"ID":"928df61d-6160-4586-acb9-d13cd90c1a72","Type":"ContainerStarted","Data":"a6a0a450f90968dc5e5cecda9a839ff4bf018cc2e5a7e8c0904a0b6bf7fa7d7a"}
Oct 30 21:01:31.145552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:31.145515    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hsrbr" event={"ID":"928df61d-6160-4586-acb9-d13cd90c1a72","Type":"ContainerStarted","Data":"5c6b1ef7fb0c7eb62032a4dd288bb03d28b5336978b4c377194ad92dbbd2bba4"}
Oct 30 21:01:31.146053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:31.145568    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:31.147032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:31.147010    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-shzqw" event={"ID":"a5946a39-1450-4d88-aae5-6f76b062fafb","Type":"ContainerStarted","Data":"ae4bf974d275b0489646b2985a49284454acf6bc3f82b87642cb33e64d7f1170"}
Oct 30 21:01:31.684215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:31.684178    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:31.684430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:31.684239    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:31.684430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:31.684273    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:31.684430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:31.684308    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:31.684430 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.684359    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:35.684336782 +0000 UTC m=+8.730895931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : configmap references non-existent config key: service-ca.crt
Oct 30 21:01:31.684430 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.684409    4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:01:31.684430 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.684412    4918 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:01:31.684430 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.684415    4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:31.684821 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.684441    4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc99cbd8-gdhf8: secret "image-registry-tls" not found
Oct 30 21:01:31.684821 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.684460    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs podName:7c27677e-bbe2-4457-b205-1faae8635cb0 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:35.684448461 +0000 UTC m=+8.731007585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs") pod "router-default-6d5fbcf987-vhd7f" (UID: "7c27677e-bbe2-4457-b205-1faae8635cb0") : secret "router-metrics-certs-default" not found
Oct 30 21:01:31.684821 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.684473    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls podName:72689c92-965b-4dfc-9f3c-4b68660f6e53 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:35.684467363 +0000 UTC m=+8.731026487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fxrz8" (UID: "72689c92-965b-4dfc-9f3c-4b68660f6e53") : secret "samples-operator-tls" not found
Oct 30 21:01:31.684821 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.684483    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls podName:ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:35.68447795 +0000 UTC m=+8.731037073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls") pod "image-registry-67cc99cbd8-gdhf8" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7") : secret "image-registry-tls" not found
Oct 30 21:01:31.785414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:31.785359    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:31.785585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:31.785438    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:31.785585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:31.785480    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:31.785585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:31.785509    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:31.785585 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.785511    4918 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:01:31.785585 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.785577    4918 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:01:31.785585 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.785579    4918 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:01:31.785902 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.785588    4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:31.785902 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.785595    4918 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67bd499c65-7dz4d: secret "image-registry-tls" not found
Oct 30 21:01:31.785902 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.785588    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert podName:148c1862-34bb-4b75-af99-96c64f2058b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:35.785568566 +0000 UTC m=+8.832127703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-22hfw" (UID: "148c1862-34bb-4b75-af99-96c64f2058b8") : secret "networking-console-plugin-cert" not found
Oct 30 21:01:31.785902 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.785672    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs podName:d80303a1-1861-4958-a037-891078b48456 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:35.785659407 +0000 UTC m=+8.832218531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs") pod "network-metrics-daemon-xppxb" (UID: "d80303a1-1861-4958-a037-891078b48456") : secret "metrics-daemon-secret" not found
Oct 30 21:01:31.785902 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.785682    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:35.785676522 +0000 UTC m=+8.832235645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:31.785902 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:31.785698    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls podName:56aaf442-dd68-4562-bc03-11ba5d35c28e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:35.785689197 +0000 UTC m=+8.832248320 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls") pod "image-registry-67bd499c65-7dz4d" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e") : secret "image-registry-tls" not found
Oct 30 21:01:32.017887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:32.017860    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:32.152185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:32.152153    4918 generic.go:356] "Generic (PLEG): container finished" podID="d4a91f73-8795-4a9d-a1cc-55ed355c0312" containerID="6e14fc5c841c51e63aa316c6956c7b8a70f535405321a4562b21b7f11df9b71c" exitCode=0
Oct 30 21:01:32.152602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:32.152229    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wg2wk" event={"ID":"d4a91f73-8795-4a9d-a1cc-55ed355c0312","Type":"ContainerDied","Data":"6e14fc5c841c51e63aa316c6956c7b8a70f535405321a4562b21b7f11df9b71c"}
Oct 30 21:01:32.152673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:32.152660    4918 scope.go:122] "RemoveContainer" containerID="6e14fc5c841c51e63aa316c6956c7b8a70f535405321a4562b21b7f11df9b71c"
Oct 30 21:01:33.157283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:33.157244    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wg2wk" event={"ID":"d4a91f73-8795-4a9d-a1cc-55ed355c0312","Type":"ContainerStarted","Data":"5caa65c08b28a1a622a1c3421e14b921b50d3918d34b9b392caf42201e7e8c32"}
Oct 30 21:01:33.867827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:33.867786    4918 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:33.868191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:33.868176    4918 scope.go:122] "RemoveContainer" containerID="ebe9de9c294b9d969f5d863ce15320b42c635db3b7988e65e196c4ef7db4db65"
Oct 30 21:01:33.868385 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:33.868365    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-22c7r_openshift-console-operator(94385c98-9c6d-4fc1-9c48-38c9e5a95447)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r" podUID="94385c98-9c6d-4fc1-9c48-38c9e5a95447"
Oct 30 21:01:35.725619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.725589    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:35.726031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.725647    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:35.726031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.725684    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:35.726031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.725737    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:35.726031 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:35.725845    4918 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:01:35.726031 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:35.725927    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls podName:72689c92-965b-4dfc-9f3c-4b68660f6e53 nodeName:}" failed. No retries permitted until 2026-10-30 21:01:43.725907223 +0000 UTC m=+16.772466362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fxrz8" (UID: "72689c92-965b-4dfc-9f3c-4b68660f6e53") : secret "samples-operator-tls" not found
Oct 30 21:01:35.726324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.726303    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c27677e-bbe2-4457-b205-1faae8635cb0-service-ca-bundle\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:35.728225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.728202    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"image-registry-67cc99cbd8-gdhf8\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") " pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:35.728318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.728276    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c27677e-bbe2-4457-b205-1faae8635cb0-metrics-certs\") pod \"router-default-6d5fbcf987-vhd7f\" (UID: \"7c27677e-bbe2-4457-b205-1faae8635cb0\") " pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:35.826967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.826932    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:35.827115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.826976    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:35.827115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.827009    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:35.827115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.827049    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:35.827223 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:35.827169    4918 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:35.827262 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:35.827237    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls podName:72f357cb-3592-424c-9391-9058d0821d5e nodeName:}" failed. No retries permitted until 2026-10-30 21:01:43.827217722 +0000 UTC m=+16.873776851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ltjmm" (UID: "72f357cb-3592-424c-9391-9058d0821d5e") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:01:35.827439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.827376    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:35.829293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.829272    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:35.829531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.829509    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d80303a1-1861-4958-a037-891078b48456-metrics-certs\") pod \"network-metrics-daemon-xppxb\" (UID: \"d80303a1-1861-4958-a037-891078b48456\") " pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:35.829629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.829611    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"image-registry-67bd499c65-7dz4d\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") " pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:35.829702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.829680    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/148c1862-34bb-4b75-af99-96c64f2058b8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-22hfw\" (UID: \"148c1862-34bb-4b75-af99-96c64f2058b8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:35.833327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.833308    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xppxb"
Oct 30 21:01:35.833418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.833385    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw"
Oct 30 21:01:35.837292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:35.837274    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:36.122103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:36.122034    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d5fbcf987-vhd7f"]
Oct 30 21:01:36.126011 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:36.125972    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c27677e_bbe2_4457_b205_1faae8635cb0.slice/crio-881cb299f657f38b0eab5eaa4af79337f696007cf9f05ae62cc9bfe9a3103e33 WatchSource:0}: Error finding container 881cb299f657f38b0eab5eaa4af79337f696007cf9f05ae62cc9bfe9a3103e33: Status 404 returned error can't find the container with id 881cb299f657f38b0eab5eaa4af79337f696007cf9f05ae62cc9bfe9a3103e33
Oct 30 21:01:36.127200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:36.127183    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67bd499c65-7dz4d"]
Oct 30 21:01:36.129035 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:36.129012    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56aaf442_dd68_4562_bc03_11ba5d35c28e.slice/crio-1d8fe08d1087a3d9bbac0724728ec88a7a6e6dfee798dab877e3274fe64e2a24 WatchSource:0}: Error finding container 1d8fe08d1087a3d9bbac0724728ec88a7a6e6dfee798dab877e3274fe64e2a24: Status 404 returned error can't find the container with id 1d8fe08d1087a3d9bbac0724728ec88a7a6e6dfee798dab877e3274fe64e2a24
Oct 30 21:01:36.162049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:36.162022    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67cc99cbd8-gdhf8"]
Oct 30 21:01:36.163724 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:36.163696    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccabbcd9_05fe_4463_b4b1_53b2d7f66fe7.slice/crio-1a1c829a19c0305dba231bca44a0ebaffd51f6420d428b6f6afd1cc735f6df39 WatchSource:0}: Error finding container 1a1c829a19c0305dba231bca44a0ebaffd51f6420d428b6f6afd1cc735f6df39: Status 404 returned error can't find the container with id 1a1c829a19c0305dba231bca44a0ebaffd51f6420d428b6f6afd1cc735f6df39
Oct 30 21:01:36.168618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:36.168079    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67bd499c65-7dz4d" event={"ID":"56aaf442-dd68-4562-bc03-11ba5d35c28e","Type":"ContainerStarted","Data":"1d8fe08d1087a3d9bbac0724728ec88a7a6e6dfee798dab877e3274fe64e2a24"}
Oct 30 21:01:36.169804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:36.169768    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d5fbcf987-vhd7f" event={"ID":"7c27677e-bbe2-4457-b205-1faae8635cb0","Type":"ContainerStarted","Data":"881cb299f657f38b0eab5eaa4af79337f696007cf9f05ae62cc9bfe9a3103e33"}
Oct 30 21:01:36.186484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:36.186258    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xppxb"]
Oct 30 21:01:36.199081 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:36.199029    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148c1862_34bb_4b75_af99_96c64f2058b8.slice/crio-34baf5ca702fb0f07075162684b01b405f23e1c908c2c4c258353e82d3297d9e WatchSource:0}: Error finding container 34baf5ca702fb0f07075162684b01b405f23e1c908c2c4c258353e82d3297d9e: Status 404 returned error can't find the container with id 34baf5ca702fb0f07075162684b01b405f23e1c908c2c4c258353e82d3297d9e
Oct 30 21:01:36.211896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:36.211873    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-22hfw"]
Oct 30 21:01:37.175311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:37.175241    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67bd499c65-7dz4d" event={"ID":"56aaf442-dd68-4562-bc03-11ba5d35c28e","Type":"ContainerStarted","Data":"f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee"}
Oct 30 21:01:37.177641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:37.177612    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw" event={"ID":"148c1862-34bb-4b75-af99-96c64f2058b8","Type":"ContainerStarted","Data":"34baf5ca702fb0f07075162684b01b405f23e1c908c2c4c258353e82d3297d9e"}
Oct 30 21:01:37.178974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:37.178950    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xppxb" event={"ID":"d80303a1-1861-4958-a037-891078b48456","Type":"ContainerStarted","Data":"5229663f7bcfc0164ba92d74cd742c119f6f913646b4bd59322f0cb9bbc1dc26"}
Oct 30 21:01:37.180615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:37.180592    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8" event={"ID":"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7","Type":"ContainerStarted","Data":"dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a"}
Oct 30 21:01:37.180751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:37.180634    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8" event={"ID":"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7","Type":"ContainerStarted","Data":"1a1c829a19c0305dba231bca44a0ebaffd51f6420d428b6f6afd1cc735f6df39"}
Oct 30 21:01:37.182261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:37.182240    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d5fbcf987-vhd7f" event={"ID":"7c27677e-bbe2-4457-b205-1faae8635cb0","Type":"ContainerStarted","Data":"3bb785d58889b15f0c36c45825a9cd7e24319703f0f7a909729668e18587b98b"}
Oct 30 21:01:37.403768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:37.403742    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:37.403958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:37.403947    4918 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:01:37.679810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:37.679794    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:37.680185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:37.680168    4918 scope.go:122] "RemoveContainer" containerID="ebe9de9c294b9d969f5d863ce15320b42c635db3b7988e65e196c4ef7db4db65"
Oct 30 21:01:37.680421 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:37.680376    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-22c7r_openshift-console-operator(94385c98-9c6d-4fc1-9c48-38c9e5a95447)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r" podUID="94385c98-9c6d-4fc1-9c48-38c9e5a95447"
Oct 30 21:01:37.827940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:37.827910    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:37.830434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:37.830414    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:38.187669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:38.187631    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-22hfw" event={"ID":"148c1862-34bb-4b75-af99-96c64f2058b8","Type":"ContainerStarted","Data":"8a5a8ab6dbf5c1790d537b299bc1e02f25c01ce7008c8181d2baee9cada34b49"}
Oct 30 21:01:38.189564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:38.189547    4918 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:01:38.189686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:38.189586    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xppxb" event={"ID":"d80303a1-1861-4958-a037-891078b48456","Type":"ContainerStarted","Data":"3f478303e3c1f2c3be3e7c3c00a896ffaeeeb5b1ab280b212e54855c580dcaab"}
Oct 30 21:01:38.189686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:38.189636    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xppxb" event={"ID":"d80303a1-1861-4958-a037-891078b48456","Type":"ContainerStarted","Data":"5f895a3e8d390902af727fd545856adcfad49d0c247b636859fa6ff7b85c4cc2"}
Oct 30 21:01:38.189686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:38.189674    4918 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:01:39.071044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:39.070620    4918 scope.go:122] "RemoveContainer" containerID="ebe9de9c294b9d969f5d863ce15320b42c635db3b7988e65e196c4ef7db4db65"
Oct 30 21:01:39.194630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:39.194599    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:01:39.195076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:39.194746    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r" event={"ID":"94385c98-9c6d-4fc1-9c48-38c9e5a95447","Type":"ContainerStarted","Data":"47bab60daebdcdf413ac714d0e0b87deade194221867ef2e42aefb7e665f5e5f"}
Oct 30 21:01:39.195076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:39.194790    4918 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:01:39.195384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:39.195369    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:39.447151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:39.447050    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-22c7r"
Oct 30 21:01:41.154991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:41.154963    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hsrbr"
Oct 30 21:01:41.803903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:41.803870    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:43.795564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:43.795522    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:43.798094 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:43.798067    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/72689c92-965b-4dfc-9f3c-4b68660f6e53-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fxrz8\" (UID: \"72689c92-965b-4dfc-9f3c-4b68660f6e53\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:43.896241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:43.896208    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:43.898561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:43.898543    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72f357cb-3592-424c-9391-9058d0821d5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ltjmm\" (UID: \"72f357cb-3592-424c-9391-9058d0821d5e\") " pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:43.924111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:43.924086    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:43.924233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:43.924219    4918 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:01:43.925360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:43.925341    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6d5fbcf987-vhd7f"
Oct 30 21:01:43.927261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:43.927243    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"
Oct 30 21:01:43.945605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:43.945581    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ltjmm"
Oct 30 21:01:44.203718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:44.203650    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8"]
Oct 30 21:01:44.229454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:44.229379    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ltjmm"]
Oct 30 21:01:44.234442 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:44.234416    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f357cb_3592_424c_9391_9058d0821d5e.slice/crio-82f7a0bbdd6ec5fbe3eea26651a4cb6b2b177098c3d1eb8ce3d9047c6057b638 WatchSource:0}: Error finding container 82f7a0bbdd6ec5fbe3eea26651a4cb6b2b177098c3d1eb8ce3d9047c6057b638: Status 404 returned error can't find the container with id 82f7a0bbdd6ec5fbe3eea26651a4cb6b2b177098c3d1eb8ce3d9047c6057b638
Oct 30 21:01:44.562033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:44.562001    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:45.223465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:45.223431    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltjmm" event={"ID":"72f357cb-3592-424c-9391-9058d0821d5e","Type":"ContainerStarted","Data":"2de746de4fdbe02e9034e4cb572a4d0b7a9a12f43769750b80e9ae4b398fd6c2"}
Oct 30 21:01:45.223984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:45.223505    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltjmm" event={"ID":"72f357cb-3592-424c-9391-9058d0821d5e","Type":"ContainerStarted","Data":"5281fccfd9eccbfaa50cd4a3dff1107a53be3584e6a915cd1718a0f854732548"}
Oct 30 21:01:45.223984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:45.223532    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltjmm" event={"ID":"72f357cb-3592-424c-9391-9058d0821d5e","Type":"ContainerStarted","Data":"82f7a0bbdd6ec5fbe3eea26651a4cb6b2b177098c3d1eb8ce3d9047c6057b638"}
Oct 30 21:01:45.224865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:45.224824    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8" event={"ID":"72689c92-965b-4dfc-9f3c-4b68660f6e53","Type":"ContainerStarted","Data":"b7a362ee9f4fd7f44626d56a2e691b13be272c490dc0f3a1959d2faf31ccf7f5"}
Oct 30 21:01:45.407418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:45.406555    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67cc99cbd8-gdhf8"]
Oct 30 21:01:46.232792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:46.232695    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8" event={"ID":"72689c92-965b-4dfc-9f3c-4b68660f6e53","Type":"ContainerStarted","Data":"fb1c3e2c77be22bb35647372b9be3f5d14ad1ddf40e1debdeecb900b8da71cac"}
Oct 30 21:01:46.232792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:46.232755    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fxrz8" event={"ID":"72689c92-965b-4dfc-9f3c-4b68660f6e53","Type":"ContainerStarted","Data":"21d09125e75fed877d8684e327a0cb24ebb088f467c02fd7fa4c7fcbaa908821"}
Oct 30 21:01:46.844448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:46.844363    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8qk"
Oct 30 21:01:47.161778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:47.161705    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hsrbr_928df61d-6160-4586-acb9-d13cd90c1a72/dns/0.log"
Oct 30 21:01:47.238579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:47.238541    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ltjmm" event={"ID":"72f357cb-3592-424c-9391-9058d0821d5e","Type":"ContainerStarted","Data":"fc0920a9610d3f2844ad815d142bfbcfe46d288e36cce88ef7d6888b080bad3e"}
Oct 30 21:01:47.361628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:47.361604    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hsrbr_928df61d-6160-4586-acb9-d13cd90c1a72/kube-rbac-proxy/0.log"
Oct 30 21:01:47.762289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:47.762262    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q7r7r_082e6a92-3eef-423d-859e-405adb01c3a1/dns-node-resolver/0.log"
Oct 30 21:01:48.362490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:48.362460    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67bd499c65-7dz4d_56aaf442-dd68-4562-bc03-11ba5d35c28e/registry/0.log"
Oct 30 21:01:48.562842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:48.562799    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67cc99cbd8-gdhf8_ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/registry/0.log"
Oct 30 21:01:48.961856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:48.961822    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rzn4z_829cdf21-2065-4e39-902d-7b857a2d74bd/node-ca/0.log"
Oct 30 21:01:49.362333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:49.362296    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d5fbcf987-vhd7f_7c27677e-bbe2-4457-b205-1faae8635cb0/router/0.log"
Oct 30 21:01:49.762683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:49.762657    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xm2j4_c736b662-1f6a-4984-b82a-9d3b455ee18f/serve-healthcheck-canary/0.log"
Oct 30 21:01:51.808499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:51.808468    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:01:54.553937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.553907    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rvml7"]
Oct 30 21:01:54.587565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.587533    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.638138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.638113    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:01:54.638138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.638119    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:01:54.638346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.638196    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:01:54.638346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.638298    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:01:54.638446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.638385    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4f5fg\""
Oct 30 21:01:54.686897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.686860    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-textfile\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.687055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.686900    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-tls\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.687055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.686928    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.687133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.687046    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ef4d880-5d0a-4c18-9e26-e1819279529f-root\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.687133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.687109    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ef4d880-5d0a-4c18-9e26-e1819279529f-metrics-client-ca\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.687203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.687132    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.687203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.687151    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ef4d880-5d0a-4c18-9e26-e1819279529f-sys\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.687203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.687171    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjlvz\" (UniqueName: \"kubernetes.io/projected/4ef4d880-5d0a-4c18-9e26-e1819279529f-kube-api-access-kjlvz\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.687313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.687298    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-wtmp\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.784541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.784510    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t48nt"
Oct 30 21:01:54.788186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788160    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjlvz\" (UniqueName: \"kubernetes.io/projected/4ef4d880-5d0a-4c18-9e26-e1819279529f-kube-api-access-kjlvz\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.788311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788192    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-wtmp\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.788311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788222    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-textfile\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.788311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788239    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-tls\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.788573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788365    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-wtmp\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.788573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788364    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.788573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788466    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ef4d880-5d0a-4c18-9e26-e1819279529f-root\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.788573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788542    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ef4d880-5d0a-4c18-9e26-e1819279529f-metrics-client-ca\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.788573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788569    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.788826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788580    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ef4d880-5d0a-4c18-9e26-e1819279529f-root\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.788826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788600    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ef4d880-5d0a-4c18-9e26-e1819279529f-sys\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.788826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788654    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-textfile\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.788826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788683    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ef4d880-5d0a-4c18-9e26-e1819279529f-sys\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.789034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.788994    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.789117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.789095    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ef4d880-5d0a-4c18-9e26-e1819279529f-metrics-client-ca\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.790825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.790797    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-tls\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.790922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.790862    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ef4d880-5d0a-4c18-9e26-e1819279529f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.808881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.808823    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjlvz\" (UniqueName: \"kubernetes.io/projected/4ef4d880-5d0a-4c18-9e26-e1819279529f-kube-api-access-kjlvz\") pod \"node-exporter-rvml7\" (UID: \"4ef4d880-5d0a-4c18-9e26-e1819279529f\") " pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.938232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:54.938204    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rvml7"
Oct 30 21:01:54.979141 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:01:54.979116    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef4d880_5d0a_4c18_9e26_e1819279529f.slice/crio-3e475fd784681a7486fbb678b7d28f97394706f3a50dc0fe9554dbab4eaa5610 WatchSource:0}: Error finding container 3e475fd784681a7486fbb678b7d28f97394706f3a50dc0fe9554dbab4eaa5610: Status 404 returned error can't find the container with id 3e475fd784681a7486fbb678b7d28f97394706f3a50dc0fe9554dbab4eaa5610
Oct 30 21:01:55.273968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:55.273935    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvml7" event={"ID":"4ef4d880-5d0a-4c18-9e26-e1819279529f","Type":"ContainerStarted","Data":"3e475fd784681a7486fbb678b7d28f97394706f3a50dc0fe9554dbab4eaa5610"}
Oct 30 21:01:55.412259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:55.412233    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:01:57.287966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:57.287927    4918 generic.go:356] "Generic (PLEG): container finished" podID="4ef4d880-5d0a-4c18-9e26-e1819279529f" containerID="a133fb4cd01f1b7b84a95f4ba0ac869394c80ae8a0e0329afbcf8d4fc0a72574" exitCode=0
Oct 30 21:01:57.288486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:57.287980    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvml7" event={"ID":"4ef4d880-5d0a-4c18-9e26-e1819279529f","Type":"ContainerDied","Data":"a133fb4cd01f1b7b84a95f4ba0ac869394c80ae8a0e0329afbcf8d4fc0a72574"}
Oct 30 21:01:58.294353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:58.294315    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvml7" event={"ID":"4ef4d880-5d0a-4c18-9e26-e1819279529f","Type":"ContainerStarted","Data":"7f9a76c170b8a9a8ac331cca6d44ec8e5cbce14f6b1b22e54aa748d765d34758"}
Oct 30 21:01:58.294786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:58.294373    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvml7" event={"ID":"4ef4d880-5d0a-4c18-9e26-e1819279529f","Type":"ContainerStarted","Data":"9e41c9592e0b9958f42f6e6e294cc52c4b035f6e893c5b61e42ad4790f13d9dd"}
Oct 30 21:01:58.315936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:01:58.315887    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rvml7" podStartSLOduration=2.645350455 podStartE2EDuration="4.315872752s" totalImagesPullingTime="1.583607681s" totalInitContainerRuntime="86.914616ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:58.314767353 +0000 UTC m=+31.361326498" watchObservedRunningTime="2026-10-30 21:01:58.315872752 +0000 UTC m=+31.362431896"
Oct 30 21:01:59.157948 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:01:59.157911    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:02:00.994721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:00.994686    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6slln"
Oct 30 21:02:08.980860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:08.980774    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67bd499c65-7dz4d"]
Oct 30 21:02:10.433609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.433562    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8" podUID="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" containerName="registry" containerID="cri-o://dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a" gracePeriod=30
Oct 30 21:02:10.694730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.694675    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:02:10.719183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.719153    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-trusted-ca\") pod \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") "
Oct 30 21:02:10.719347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.719196    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-kube-api-access-m9p2b\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-kube-api-access-m9p2b\") pod \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") "
Oct 30 21:02:10.719347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.719311    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-installation-pull-secrets\") pod \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") "
Oct 30 21:02:10.719465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.719361    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-image-registry-private-configuration\") pod \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") "
Oct 30 21:02:10.719465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.719408    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") pod \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") "
Oct 30 21:02:10.719569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.719478    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-bound-sa-token\") pod \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") "
Oct 30 21:02:10.719569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.719516    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-trusted-ca" pod "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:10.719676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.719575    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-certificates\") pod \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") "
Oct 30 21:02:10.719676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.719607    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-ca-trust-extracted\") pod \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\" (UID: \"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7\") "
Oct 30 21:02:10.721448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.719855    4918 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-trusted-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:10.721448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.720149    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-certificates" pod "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:10.722730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.722679    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-bound-sa-token" pod "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:10.722827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.722786    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-kube-api-access-m9p2b" pod "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7"). InnerVolumeSpecName "kube-api-access-m9p2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:10.723052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.723026    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-image-registry-private-configuration" pod "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:10.723183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.723124    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-installation-pull-secrets" pod "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:10.724678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.724652    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls" pod "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:10.728147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.728127    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-ca-trust-extracted" pod "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" (UID: "ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:10.821037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.821007    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9p2b\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-kube-api-access-m9p2b\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:10.821037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.821031    4918 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-installation-pull-secrets\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:10.821037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.821041    4918 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-image-registry-private-configuration\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:10.821253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.821052    4918 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:10.821253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.821061    4918 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-bound-sa-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:10.821253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.821069    4918 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-registry-certificates\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:10.821253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:10.821079    4918 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7-ca-trust-extracted\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:11.344796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:11.344705    4918 generic.go:356] "Generic (PLEG): container finished" podID="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" containerID="dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a" exitCode=0
Oct 30 21:02:11.344796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:11.344757    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8"
Oct 30 21:02:11.345015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:11.344798    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8" event={"ID":"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7","Type":"ContainerDied","Data":"dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a"}
Oct 30 21:02:11.345015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:11.344848    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67cc99cbd8-gdhf8" event={"ID":"ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7","Type":"ContainerDied","Data":"1a1c829a19c0305dba231bca44a0ebaffd51f6420d428b6f6afd1cc735f6df39"}
Oct 30 21:02:11.345015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:11.344875    4918 scope.go:122] "RemoveContainer" containerID="dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a"
Oct 30 21:02:11.353234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:11.353217    4918 scope.go:122] "RemoveContainer" containerID="dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a"
Oct 30 21:02:11.353533 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:02:11.353513    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a\": container with ID starting with dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a not found: ID does not exist" containerID="dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a"
Oct 30 21:02:11.353606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:11.353548    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a"} err="failed to get container status \"dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a\": rpc error: code = NotFound desc = could not find container \"dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a\": container with ID starting with dcbc2d9b0057d69370fc7d2422c77f527f5bb839418065035937a0811712235a not found: ID does not exist"
Oct 30 21:02:11.371415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:11.371377    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67cc99cbd8-gdhf8"]
Oct 30 21:02:11.378814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:11.378794    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67cc99cbd8-gdhf8"]
Oct 30 21:02:13.074055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:13.074018    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" path="/var/lib/kubelet/pods/ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7/volumes"
Oct 30 21:02:29.158016 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:02:29.157980    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:02:34.001170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.001089    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67bd499c65-7dz4d" podUID="56aaf442-dd68-4562-bc03-11ba5d35c28e" containerName="registry" containerID="cri-o://f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee" gracePeriod=30
Oct 30 21:02:34.232261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.232235    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:02:34.301615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.301543    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") pod \"56aaf442-dd68-4562-bc03-11ba5d35c28e\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") "
Oct 30 21:02:34.301615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.301588    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56aaf442-dd68-4562-bc03-11ba5d35c28e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56aaf442-dd68-4562-bc03-11ba5d35c28e-ca-trust-extracted\") pod \"56aaf442-dd68-4562-bc03-11ba5d35c28e\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") "
Oct 30 21:02:34.301777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.301618    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-image-registry-private-configuration\") pod \"56aaf442-dd68-4562-bc03-11ba5d35c28e\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") "
Oct 30 21:02:34.301777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.301736    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-certificates\") pod \"56aaf442-dd68-4562-bc03-11ba5d35c28e\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") "
Oct 30 21:02:34.301854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.301807    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-trusted-ca\") pod \"56aaf442-dd68-4562-bc03-11ba5d35c28e\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") "
Oct 30 21:02:34.302250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.301966    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-installation-pull-secrets\") pod \"56aaf442-dd68-4562-bc03-11ba5d35c28e\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") "
Oct 30 21:02:34.302250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.302012    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-bound-sa-token\") pod \"56aaf442-dd68-4562-bc03-11ba5d35c28e\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") "
Oct 30 21:02:34.302250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.302094    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-kube-api-access-k9jdg\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-kube-api-access-k9jdg\") pod \"56aaf442-dd68-4562-bc03-11ba5d35c28e\" (UID: \"56aaf442-dd68-4562-bc03-11ba5d35c28e\") "
Oct 30 21:02:34.302250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.302167    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-trusted-ca" pod "56aaf442-dd68-4562-bc03-11ba5d35c28e" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:34.302250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.302177    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-certificates" pod "56aaf442-dd68-4562-bc03-11ba5d35c28e" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:02:34.302594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.302348    4918 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-certificates\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.302594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.302371    4918 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56aaf442-dd68-4562-bc03-11ba5d35c28e-trusted-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.304506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.304474    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls" pod "56aaf442-dd68-4562-bc03-11ba5d35c28e" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:34.304707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.304684    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-image-registry-private-configuration" pod "56aaf442-dd68-4562-bc03-11ba5d35c28e" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:34.304970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.304939    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-kube-api-access-k9jdg" pod "56aaf442-dd68-4562-bc03-11ba5d35c28e" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e"). InnerVolumeSpecName "kube-api-access-k9jdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:34.305071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.304987    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-installation-pull-secrets" pod "56aaf442-dd68-4562-bc03-11ba5d35c28e" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:02:34.305071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.305005    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-bound-sa-token" pod "56aaf442-dd68-4562-bc03-11ba5d35c28e" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:02:34.309138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.309121    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56aaf442-dd68-4562-bc03-11ba5d35c28e-ca-trust-extracted" pod "56aaf442-dd68-4562-bc03-11ba5d35c28e" (UID: "56aaf442-dd68-4562-bc03-11ba5d35c28e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:02:34.403680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.403654    4918 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/56aaf442-dd68-4562-bc03-11ba5d35c28e-ca-trust-extracted\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.403680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.403678    4918 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-image-registry-private-configuration\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.403838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.403690    4918 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/56aaf442-dd68-4562-bc03-11ba5d35c28e-installation-pull-secrets\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.403838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.403700    4918 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-bound-sa-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.403838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.403708    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9jdg\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-kube-api-access-k9jdg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.403838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.403717    4918 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/56aaf442-dd68-4562-bc03-11ba5d35c28e-registry-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:02:34.440971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.440940    4918 generic.go:356] "Generic (PLEG): container finished" podID="56aaf442-dd68-4562-bc03-11ba5d35c28e" containerID="f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee" exitCode=0
Oct 30 21:02:34.441081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.440998    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67bd499c65-7dz4d"
Oct 30 21:02:34.441081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.441020    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67bd499c65-7dz4d" event={"ID":"56aaf442-dd68-4562-bc03-11ba5d35c28e","Type":"ContainerDied","Data":"f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee"}
Oct 30 21:02:34.441081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.441058    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67bd499c65-7dz4d" event={"ID":"56aaf442-dd68-4562-bc03-11ba5d35c28e","Type":"ContainerDied","Data":"1d8fe08d1087a3d9bbac0724728ec88a7a6e6dfee798dab877e3274fe64e2a24"}
Oct 30 21:02:34.441081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.441076    4918 scope.go:122] "RemoveContainer" containerID="f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee"
Oct 30 21:02:34.449603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.449588    4918 scope.go:122] "RemoveContainer" containerID="f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee"
Oct 30 21:02:34.449853 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:02:34.449837    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee\": container with ID starting with f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee not found: ID does not exist" containerID="f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee"
Oct 30 21:02:34.449893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.449863    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee"} err="failed to get container status \"f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee\": rpc error: code = NotFound desc = could not find container \"f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee\": container with ID starting with f0ecff2eb5d8f70039d526d59c2d1a9a7164a8e19e851b9f03db2b4b189267ee not found: ID does not exist"
Oct 30 21:02:34.461950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.461921    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67bd499c65-7dz4d"]
Oct 30 21:02:34.467341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:34.467319    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67bd499c65-7dz4d"]
Oct 30 21:02:35.074159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:35.074130    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56aaf442-dd68-4562-bc03-11ba5d35c28e" path="/var/lib/kubelet/pods/56aaf442-dd68-4562-bc03-11ba5d35c28e/volumes"
Oct 30 21:02:42.474365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:42.474335    4918 generic.go:356] "Generic (PLEG): container finished" podID="3f89330a-b946-4b7d-8024-5ed5825719e3" containerID="7ca531b22ea9acafc81169267837908dce407429f090188af535124265eba246" exitCode=0
Oct 30 21:02:42.474785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:42.474379    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8" event={"ID":"3f89330a-b946-4b7d-8024-5ed5825719e3","Type":"ContainerDied","Data":"7ca531b22ea9acafc81169267837908dce407429f090188af535124265eba246"}
Oct 30 21:02:42.474785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:42.474738    4918 scope.go:122] "RemoveContainer" containerID="7ca531b22ea9acafc81169267837908dce407429f090188af535124265eba246"
Oct 30 21:02:43.480233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:43.480200    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-2djc8" event={"ID":"3f89330a-b946-4b7d-8024-5ed5825719e3","Type":"ContainerStarted","Data":"8f05a9c5ba94734f1a56044cc9f985a96ebe23faeaa2dde3321a676034a3ca85"}
Oct 30 21:02:52.517250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:52.517212    4918 generic.go:356] "Generic (PLEG): container finished" podID="d4a91f73-8795-4a9d-a1cc-55ed355c0312" containerID="5caa65c08b28a1a622a1c3421e14b921b50d3918d34b9b392caf42201e7e8c32" exitCode=0
Oct 30 21:02:52.517640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:52.517285    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wg2wk" event={"ID":"d4a91f73-8795-4a9d-a1cc-55ed355c0312","Type":"ContainerDied","Data":"5caa65c08b28a1a622a1c3421e14b921b50d3918d34b9b392caf42201e7e8c32"}
Oct 30 21:02:52.517640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:52.517342    4918 scope.go:122] "RemoveContainer" containerID="6e14fc5c841c51e63aa316c6956c7b8a70f535405321a4562b21b7f11df9b71c"
Oct 30 21:02:52.517640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:52.517623    4918 scope.go:122] "RemoveContainer" containerID="5caa65c08b28a1a622a1c3421e14b921b50d3918d34b9b392caf42201e7e8c32"
Oct 30 21:02:52.517832 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:02:52.517815    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-wg2wk_openshift-insights(d4a91f73-8795-4a9d-a1cc-55ed355c0312)\"" pod="openshift-insights/insights-operator-7c4986d449-wg2wk" podUID="d4a91f73-8795-4a9d-a1cc-55ed355c0312"
Oct 30 21:02:57.538921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:57.538891    4918 generic.go:356] "Generic (PLEG): container finished" podID="722bd8ce-7062-4e97-a0e8-0984c73c2128" containerID="cd246e75275ecc57da3457502f755bca35aef04edd940fb4a38eaec779eb7c61" exitCode=0
Oct 30 21:02:57.539294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:57.538966    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c" event={"ID":"722bd8ce-7062-4e97-a0e8-0984c73c2128","Type":"ContainerDied","Data":"cd246e75275ecc57da3457502f755bca35aef04edd940fb4a38eaec779eb7c61"}
Oct 30 21:02:57.539294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:57.539276    4918 scope.go:122] "RemoveContainer" containerID="cd246e75275ecc57da3457502f755bca35aef04edd940fb4a38eaec779eb7c61"
Oct 30 21:02:58.544204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:02:58.544173    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9pm2c" event={"ID":"722bd8ce-7062-4e97-a0e8-0984c73c2128","Type":"ContainerStarted","Data":"89ce8ad690d107d11c99a2875cd2c4ec82a44189f76a37ccd8b2c85a19ad8a60"}
Oct 30 21:02:59.159854 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:02:59.159814    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:03:07.073009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:03:07.072978    4918 scope.go:122] "RemoveContainer" containerID="5caa65c08b28a1a622a1c3421e14b921b50d3918d34b9b392caf42201e7e8c32"
Oct 30 21:03:07.578961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:03:07.578933    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-wg2wk" event={"ID":"d4a91f73-8795-4a9d-a1cc-55ed355c0312","Type":"ContainerStarted","Data":"c711f547565d87a10c70f11246c045377e7fa1e343449f09e1770275a46c0308"}
Oct 30 21:03:29.158536 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:03:29.158493    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:03:59.158962 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:03:59.158878    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:04:29.158554 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:04:29.158509    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:04:59.158861 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:04:59.158815    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:05:23.378924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.378846    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"]
Oct 30 21:05:23.379280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.379126    4918 cpu_manager.go:414] "containerMap: removing container" podUID="56aaf442-dd68-4562-bc03-11ba5d35c28e" containerName="registry"
Oct 30 21:05:23.379280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.379135    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56aaf442-dd68-4562-bc03-11ba5d35c28e" containerName="registry"
Oct 30 21:05:23.379280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.379160    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" containerName="registry"
Oct 30 21:05:23.379280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.379166    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" containerName="registry"
Oct 30 21:05:23.379280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.379212    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="56aaf442-dd68-4562-bc03-11ba5d35c28e" containerName="registry"
Oct 30 21:05:23.379280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.379223    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccabbcd9-05fe-4463-b4b1-53b2d7f66fe7" containerName="registry"
Oct 30 21:05:23.382231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.382215    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.389829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.389810    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:05:23.390856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.390842    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:05:23.390922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.390853    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z426l\""
Oct 30 21:05:23.390922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.390865    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:05:23.390922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.390899    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:05:23.417177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.417152    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"]
Oct 30 21:05:23.452644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.452619    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.452741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.452649    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.452741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.452712    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.452819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.452764    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.452858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.452821    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.452898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.452867    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.452898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.452883    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd6c9\" (UniqueName: \"kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-kube-api-access-sd6c9\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.553514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.553486    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.553514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.553515    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd6c9\" (UniqueName: \"kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-kube-api-access-sd6c9\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.553674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.553536    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.553674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.553555    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.553674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.553582    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.553674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.553639    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.553674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.553661    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.554310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.554288    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.556075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.556047    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.556175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.556079    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.556175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.556092    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.556175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.556160    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.577321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.577299    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.578439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.578423    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd6c9\" (UniqueName: \"kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-kube-api-access-sd6c9\") pod \"istiod-openshift-gateway-7876bfcb74-vxp5t\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.693773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.693697    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:23.979749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:23.979524    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"]
Oct 30 21:05:23.981835 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:05:23.981812    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f2f8cb_44c0_40b7_959a_4ebbf3d743c0.slice/crio-e7462d7006bb0eedf67e67222da10dd6e16bcb4d6c81c086098b3fee664e473c WatchSource:0}: Error finding container e7462d7006bb0eedf67e67222da10dd6e16bcb4d6c81c086098b3fee664e473c: Status 404 returned error can't find the container with id e7462d7006bb0eedf67e67222da10dd6e16bcb4d6c81c086098b3fee664e473c
Oct 30 21:05:24.057657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:24.057626    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t" event={"ID":"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0","Type":"ContainerStarted","Data":"e7462d7006bb0eedf67e67222da10dd6e16bcb4d6c81c086098b3fee664e473c"}
Oct 30 21:05:26.859199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:26.859168    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:05:26.859503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:26.859232    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:05:27.073102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:27.073071    4918 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-vxp5t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:05:27.073267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:27.073133    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t" podUID="a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:05:27.074558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:27.074531    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t" event={"ID":"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0","Type":"ContainerStarted","Data":"28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a"}
Oct 30 21:05:27.074697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:27.074579    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:27.099288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:27.099242    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t" podStartSLOduration=1.224486952 podStartE2EDuration="4.099230334s" totalImagesPullingTime="2.874743382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:27.097080659 +0000 UTC m=+240.143639815" watchObservedRunningTime="2026-10-30 21:05:27.099230334 +0000 UTC m=+240.145789479"
Oct 30 21:05:28.074920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:28.074892    4918 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-vxp5t container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:05:28.075290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:28.074938    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t" podUID="a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:05:29.158523 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:05:29.158490    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:05:31.075037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:05:31.075005    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:05:59.161297 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:05:59.161254    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:06:16.630996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.630962    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"]
Oct 30 21:06:16.634414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.634380    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.675945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.675917    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"]
Oct 30 21:06:16.710254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.710227    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:06:16.710430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.710254    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:06:16.710430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.710264    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:06:16.710430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.710285    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:06:16.710430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.710303    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:06:16.710430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.710416    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wf7xj\""
Oct 30 21:06:16.757510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.757485    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90a8270a-47bd-4282-bcbd-ae49701d8ab0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.757710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.757516    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90a8270a-47bd-4282-bcbd-ae49701d8ab0-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.757710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.757535    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90a8270a-47bd-4282-bcbd-ae49701d8ab0-tmp\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.757710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.757607    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rbsv\" (UniqueName: \"kubernetes.io/projected/90a8270a-47bd-4282-bcbd-ae49701d8ab0-kube-api-access-8rbsv\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.757824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.757725    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90a8270a-47bd-4282-bcbd-ae49701d8ab0-manifests\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.757824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.757749    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90a8270a-47bd-4282-bcbd-ae49701d8ab0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.858580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.858550    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rbsv\" (UniqueName: \"kubernetes.io/projected/90a8270a-47bd-4282-bcbd-ae49701d8ab0-kube-api-access-8rbsv\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.858751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.858604    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90a8270a-47bd-4282-bcbd-ae49701d8ab0-manifests\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.858751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.858629    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90a8270a-47bd-4282-bcbd-ae49701d8ab0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.858751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.858671    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90a8270a-47bd-4282-bcbd-ae49701d8ab0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.858751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.858708    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90a8270a-47bd-4282-bcbd-ae49701d8ab0-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.858751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.858737    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90a8270a-47bd-4282-bcbd-ae49701d8ab0-tmp\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.859167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.859053    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90a8270a-47bd-4282-bcbd-ae49701d8ab0-manifests\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.861258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.861234    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90a8270a-47bd-4282-bcbd-ae49701d8ab0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.861376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.861329    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90a8270a-47bd-4282-bcbd-ae49701d8ab0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.861470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.861432    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90a8270a-47bd-4282-bcbd-ae49701d8ab0-tmp\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.862012 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.861964    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90a8270a-47bd-4282-bcbd-ae49701d8ab0-metrics-cert\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.872383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.872356    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rbsv\" (UniqueName: \"kubernetes.io/projected/90a8270a-47bd-4282-bcbd-ae49701d8ab0-kube-api-access-8rbsv\") pod \"opendatahub-operator-controller-manager-7bd6789b8f-s2qbx\" (UID: \"90a8270a-47bd-4282-bcbd-ae49701d8ab0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:16.948860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:16.948791    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:17.246112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:17.246090    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"]
Oct 30 21:06:17.247496 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:06:17.247465    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a8270a_47bd_4282_bcbd_ae49701d8ab0.slice/crio-443f425db319e54ce2a3625d247a611486d518cc08d356699d6f9be836870ce3 WatchSource:0}: Error finding container 443f425db319e54ce2a3625d247a611486d518cc08d356699d6f9be836870ce3: Status 404 returned error can't find the container with id 443f425db319e54ce2a3625d247a611486d518cc08d356699d6f9be836870ce3
Oct 30 21:06:17.267668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:17.267633    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx" event={"ID":"90a8270a-47bd-4282-bcbd-ae49701d8ab0","Type":"ContainerStarted","Data":"443f425db319e54ce2a3625d247a611486d518cc08d356699d6f9be836870ce3"}
Oct 30 21:06:20.283808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:20.283770    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx" event={"ID":"90a8270a-47bd-4282-bcbd-ae49701d8ab0","Type":"ContainerStarted","Data":"88be843d944092f1f2ff1886c51b3bad27736db026e0b12588809845017f8f13"}
Oct 30 21:06:21.289173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:21.289138    4918 generic.go:356] "Generic (PLEG): container finished" podID="90a8270a-47bd-4282-bcbd-ae49701d8ab0" containerID="88be843d944092f1f2ff1886c51b3bad27736db026e0b12588809845017f8f13" exitCode=0
Oct 30 21:06:21.289590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:21.289185    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx" event={"ID":"90a8270a-47bd-4282-bcbd-ae49701d8ab0","Type":"ContainerDied","Data":"88be843d944092f1f2ff1886c51b3bad27736db026e0b12588809845017f8f13"}
Oct 30 21:06:22.295580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:22.295545    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx" event={"ID":"90a8270a-47bd-4282-bcbd-ae49701d8ab0","Type":"ContainerStarted","Data":"8e898a11057354cc8cb75a69bfbcc27cd53f0fc5b310d54b7f4be5ff7b52c20e"}
Oct 30 21:06:22.295965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:22.295657    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:22.375202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:22.375151    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx" podStartSLOduration=2.457603476 podStartE2EDuration="6.375137681s" totalImagesPullingTime="3.478987562s" totalInitContainerRuntime="438.546643ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:22.372732425 +0000 UTC m=+295.419291579" watchObservedRunningTime="2026-10-30 21:06:22.375137681 +0000 UTC m=+295.421696826"
Oct 30 21:06:27.075881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:27.075854    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:06:27.076303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:27.076085    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:06:29.158707 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:06:29.158671    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:06:33.301817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:33.301784    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7bd6789b8f-s2qbx"
Oct 30 21:06:54.298071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.297996    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"]
Oct 30 21:06:54.304894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.304867    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"
Oct 30 21:06:54.317479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.317454    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"]
Oct 30 21:06:54.318817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.318799    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:06:54.318913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.318890    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 21:06:54.320221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.320202    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-t6lsf\""
Oct 30 21:06:54.457876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.457849    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t82g\" (UniqueName: \"kubernetes.io/projected/7971b944-89b9-447c-96d8-8111a389680b-kube-api-access-8t82g\") pod \"kube-auth-proxy-5978974d9d-92d2b\" (UID: \"7971b944-89b9-447c-96d8-8111a389680b\") " pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"
Oct 30 21:06:54.458025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.457891    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7971b944-89b9-447c-96d8-8111a389680b-tmp\") pod \"kube-auth-proxy-5978974d9d-92d2b\" (UID: \"7971b944-89b9-447c-96d8-8111a389680b\") " pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"
Oct 30 21:06:54.458025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.457968    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7971b944-89b9-447c-96d8-8111a389680b-tls-certs\") pod \"kube-auth-proxy-5978974d9d-92d2b\" (UID: \"7971b944-89b9-447c-96d8-8111a389680b\") " pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"
Oct 30 21:06:54.559427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.559327    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t82g\" (UniqueName: \"kubernetes.io/projected/7971b944-89b9-447c-96d8-8111a389680b-kube-api-access-8t82g\") pod \"kube-auth-proxy-5978974d9d-92d2b\" (UID: \"7971b944-89b9-447c-96d8-8111a389680b\") " pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"
Oct 30 21:06:54.559427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.559373    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7971b944-89b9-447c-96d8-8111a389680b-tmp\") pod \"kube-auth-proxy-5978974d9d-92d2b\" (UID: \"7971b944-89b9-447c-96d8-8111a389680b\") " pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"
Oct 30 21:06:54.559427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.559427    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7971b944-89b9-447c-96d8-8111a389680b-tls-certs\") pod \"kube-auth-proxy-5978974d9d-92d2b\" (UID: \"7971b944-89b9-447c-96d8-8111a389680b\") " pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"
Oct 30 21:06:54.561806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.561773    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7971b944-89b9-447c-96d8-8111a389680b-tmp\") pod \"kube-auth-proxy-5978974d9d-92d2b\" (UID: \"7971b944-89b9-447c-96d8-8111a389680b\") " pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"
Oct 30 21:06:54.562062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.562045    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7971b944-89b9-447c-96d8-8111a389680b-tls-certs\") pod \"kube-auth-proxy-5978974d9d-92d2b\" (UID: \"7971b944-89b9-447c-96d8-8111a389680b\") " pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"
Oct 30 21:06:54.568016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.567995    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t82g\" (UniqueName: \"kubernetes.io/projected/7971b944-89b9-447c-96d8-8111a389680b-kube-api-access-8t82g\") pod \"kube-auth-proxy-5978974d9d-92d2b\" (UID: \"7971b944-89b9-447c-96d8-8111a389680b\") " pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"
Oct 30 21:06:54.619729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.619705    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"
Oct 30 21:06:54.895802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.895726    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5978974d9d-92d2b"]
Oct 30 21:06:54.898914 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:06:54.898888    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7971b944_89b9_447c_96d8_8111a389680b.slice/crio-c7cffff04f2039c015f9433f1ae6049a70c8b8a246f246d2d0e36c278a444982 WatchSource:0}: Error finding container c7cffff04f2039c015f9433f1ae6049a70c8b8a246f246d2d0e36c278a444982: Status 404 returned error can't find the container with id c7cffff04f2039c015f9433f1ae6049a70c8b8a246f246d2d0e36c278a444982
Oct 30 21:06:54.900945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:54.900925    4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:06:55.421831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:55.421796    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b" event={"ID":"7971b944-89b9-447c-96d8-8111a389680b","Type":"ContainerStarted","Data":"c7cffff04f2039c015f9433f1ae6049a70c8b8a246f246d2d0e36c278a444982"}
Oct 30 21:06:57.432006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:57.431977    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b" event={"ID":"7971b944-89b9-447c-96d8-8111a389680b","Type":"ContainerStarted","Data":"ed846fe8cc274c725ef66830547c59c72ac8305378b21c88a46002055834245c"}
Oct 30 21:06:57.450190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:06:57.450147    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5978974d9d-92d2b" podStartSLOduration=1.075100302 podStartE2EDuration="3.450134717s" totalImagesPullingTime="2.375034415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:57.448765029 +0000 UTC m=+330.495324176" watchObservedRunningTime="2026-10-30 21:06:57.450134717 +0000 UTC m=+330.496693862"
Oct 30 21:06:59.159295 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:06:59.159262    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:07:12.936877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:12.936838    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-58bzk"]
Oct 30 21:07:12.940340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:12.940317    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-58bzk"
Oct 30 21:07:12.945803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:12.945778    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zlhck\""
Oct 30 21:07:12.946817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:12.946794    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-58bzk"]
Oct 30 21:07:12.947026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:12.947003    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:07:12.947101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:12.947003    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:07:13.009590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.009560    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmthb\" (UniqueName: \"kubernetes.io/projected/3cd3ef1e-da27-4bed-b37f-addadd05af18-kube-api-access-tmthb\") pod \"authorino-57f7f8659f-58bzk\" (UID: \"3cd3ef1e-da27-4bed-b37f-addadd05af18\") " pod="kuadrant-system/authorino-57f7f8659f-58bzk"
Oct 30 21:07:13.090187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.090161    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"]
Oct 30 21:07:13.093427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.093410    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.103891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.103872    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"]
Oct 30 21:07:13.104482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.104463    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 21:07:13.104565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.104486    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 21:07:13.104565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.104510    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:07:13.111009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.110983    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69e8d919-b488-4a27-854f-39202f6335ad-metrics-certs\") pod \"jobset-controller-manager-6dbd7d59bd-wgg2g\" (UID: \"69e8d919-b488-4a27-854f-39202f6335ad\") " pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.111123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.111105    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmthb\" (UniqueName: \"kubernetes.io/projected/3cd3ef1e-da27-4bed-b37f-addadd05af18-kube-api-access-tmthb\") pod \"authorino-57f7f8659f-58bzk\" (UID: \"3cd3ef1e-da27-4bed-b37f-addadd05af18\") " pod="kuadrant-system/authorino-57f7f8659f-58bzk"
Oct 30 21:07:13.111190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.111151    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/69e8d919-b488-4a27-854f-39202f6335ad-manager-config\") pod \"jobset-controller-manager-6dbd7d59bd-wgg2g\" (UID: \"69e8d919-b488-4a27-854f-39202f6335ad\") " pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.111295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.111202    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69e8d919-b488-4a27-854f-39202f6335ad-cert\") pod \"jobset-controller-manager-6dbd7d59bd-wgg2g\" (UID: \"69e8d919-b488-4a27-854f-39202f6335ad\") " pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.111295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.111228    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2pw\" (UniqueName: \"kubernetes.io/projected/69e8d919-b488-4a27-854f-39202f6335ad-kube-api-access-cg2pw\") pod \"jobset-controller-manager-6dbd7d59bd-wgg2g\" (UID: \"69e8d919-b488-4a27-854f-39202f6335ad\") " pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.120689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.120671    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmthb\" (UniqueName: \"kubernetes.io/projected/3cd3ef1e-da27-4bed-b37f-addadd05af18-kube-api-access-tmthb\") pod \"authorino-57f7f8659f-58bzk\" (UID: \"3cd3ef1e-da27-4bed-b37f-addadd05af18\") " pod="kuadrant-system/authorino-57f7f8659f-58bzk"
Oct 30 21:07:13.207763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.207672    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 21:07:13.207763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.207730    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-wz599\""
Oct 30 21:07:13.207936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.207764    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 21:07:13.212215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.212200    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69e8d919-b488-4a27-854f-39202f6335ad-metrics-certs\") pod \"jobset-controller-manager-6dbd7d59bd-wgg2g\" (UID: \"69e8d919-b488-4a27-854f-39202f6335ad\") " pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.212268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.212244    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/69e8d919-b488-4a27-854f-39202f6335ad-manager-config\") pod \"jobset-controller-manager-6dbd7d59bd-wgg2g\" (UID: \"69e8d919-b488-4a27-854f-39202f6335ad\") " pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.212370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.212357    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69e8d919-b488-4a27-854f-39202f6335ad-cert\") pod \"jobset-controller-manager-6dbd7d59bd-wgg2g\" (UID: \"69e8d919-b488-4a27-854f-39202f6335ad\") " pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.212432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.212407    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2pw\" (UniqueName: \"kubernetes.io/projected/69e8d919-b488-4a27-854f-39202f6335ad-kube-api-access-cg2pw\") pod \"jobset-controller-manager-6dbd7d59bd-wgg2g\" (UID: \"69e8d919-b488-4a27-854f-39202f6335ad\") " pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.213085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.212987    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/69e8d919-b488-4a27-854f-39202f6335ad-manager-config\") pod \"jobset-controller-manager-6dbd7d59bd-wgg2g\" (UID: \"69e8d919-b488-4a27-854f-39202f6335ad\") " pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.214757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.214730    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69e8d919-b488-4a27-854f-39202f6335ad-metrics-certs\") pod \"jobset-controller-manager-6dbd7d59bd-wgg2g\" (UID: \"69e8d919-b488-4a27-854f-39202f6335ad\") " pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.214892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.214874    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69e8d919-b488-4a27-854f-39202f6335ad-cert\") pod \"jobset-controller-manager-6dbd7d59bd-wgg2g\" (UID: \"69e8d919-b488-4a27-854f-39202f6335ad\") " pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.220910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.220888    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2pw\" (UniqueName: \"kubernetes.io/projected/69e8d919-b488-4a27-854f-39202f6335ad-kube-api-access-cg2pw\") pod \"jobset-controller-manager-6dbd7d59bd-wgg2g\" (UID: \"69e8d919-b488-4a27-854f-39202f6335ad\") " pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.259872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.259853    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-58bzk"
Oct 30 21:07:13.405336 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.405305    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:13.528027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.527978    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-58bzk"]
Oct 30 21:07:13.531047 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:07:13.531015    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cd3ef1e_da27_4bed_b37f_addadd05af18.slice/crio-99e7a3b43cf256424b5e5a714a8eeb82648d3e6f2b9bab59dc0c558205ca323b WatchSource:0}: Error finding container 99e7a3b43cf256424b5e5a714a8eeb82648d3e6f2b9bab59dc0c558205ca323b: Status 404 returned error can't find the container with id 99e7a3b43cf256424b5e5a714a8eeb82648d3e6f2b9bab59dc0c558205ca323b
Oct 30 21:07:13.677497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:13.677352    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"]
Oct 30 21:07:13.679937 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:07:13.679913    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e8d919_b488_4a27_854f_39202f6335ad.slice/crio-d3d29a180d85ba57defd61d57037c61442efb3f1a35c23601a4a763ac087e909 WatchSource:0}: Error finding container d3d29a180d85ba57defd61d57037c61442efb3f1a35c23601a4a763ac087e909: Status 404 returned error can't find the container with id d3d29a180d85ba57defd61d57037c61442efb3f1a35c23601a4a763ac087e909
Oct 30 21:07:14.498770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:14.498739    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-58bzk" event={"ID":"3cd3ef1e-da27-4bed-b37f-addadd05af18","Type":"ContainerStarted","Data":"99e7a3b43cf256424b5e5a714a8eeb82648d3e6f2b9bab59dc0c558205ca323b"}
Oct 30 21:07:14.499923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:14.499900    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g" event={"ID":"69e8d919-b488-4a27-854f-39202f6335ad","Type":"ContainerStarted","Data":"d3d29a180d85ba57defd61d57037c61442efb3f1a35c23601a4a763ac087e909"}
Oct 30 21:07:16.020751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:16.020718    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-58bzk"]
Oct 30 21:07:17.517920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:17.517881    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-58bzk" event={"ID":"3cd3ef1e-da27-4bed-b37f-addadd05af18","Type":"ContainerStarted","Data":"d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36"}
Oct 30 21:07:17.518350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:17.517907    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-58bzk" podUID="3cd3ef1e-da27-4bed-b37f-addadd05af18" containerName="authorino" containerID="cri-o://d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36" gracePeriod=30
Oct 30 21:07:17.519517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:17.519490    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g" event={"ID":"69e8d919-b488-4a27-854f-39202f6335ad","Type":"ContainerStarted","Data":"1ebf52ffe7d8abb429dce64404f1271ddcd8f7705d538132900eaf5ff1fcf119"}
Oct 30 21:07:17.519659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:17.519631    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:17.535646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:17.535604    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-58bzk" podStartSLOduration=2.198097919 podStartE2EDuration="5.535592338s" totalImagesPullingTime="3.337494419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:17.533696792 +0000 UTC m=+350.580255955" watchObservedRunningTime="2026-10-30 21:07:17.535592338 +0000 UTC m=+350.582151483"
Oct 30 21:07:17.554655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:17.554619    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g" podStartSLOduration=1.364455641 podStartE2EDuration="4.554607731s" totalImagesPullingTime="3.19015209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:17.553879812 +0000 UTC m=+350.600438971" watchObservedRunningTime="2026-10-30 21:07:17.554607731 +0000 UTC m=+350.601166904"
Oct 30 21:07:17.760816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:17.760795    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-58bzk"
Oct 30 21:07:17.849790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:17.849711    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd3ef1e-da27-4bed-b37f-addadd05af18-kube-api-access-tmthb\" (UniqueName: \"kubernetes.io/projected/3cd3ef1e-da27-4bed-b37f-addadd05af18-kube-api-access-tmthb\") pod \"3cd3ef1e-da27-4bed-b37f-addadd05af18\" (UID: \"3cd3ef1e-da27-4bed-b37f-addadd05af18\") "
Oct 30 21:07:17.851991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:17.851969    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd3ef1e-da27-4bed-b37f-addadd05af18-kube-api-access-tmthb" pod "3cd3ef1e-da27-4bed-b37f-addadd05af18" (UID: "3cd3ef1e-da27-4bed-b37f-addadd05af18"). InnerVolumeSpecName "kube-api-access-tmthb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:17.950517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:17.950487    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmthb\" (UniqueName: \"kubernetes.io/projected/3cd3ef1e-da27-4bed-b37f-addadd05af18-kube-api-access-tmthb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:18.525711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:18.525677    4918 generic.go:356] "Generic (PLEG): container finished" podID="3cd3ef1e-da27-4bed-b37f-addadd05af18" containerID="d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36" exitCode=0
Oct 30 21:07:18.525711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:18.525717    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-58bzk"
Oct 30 21:07:18.526222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:18.525761    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-58bzk" event={"ID":"3cd3ef1e-da27-4bed-b37f-addadd05af18","Type":"ContainerDied","Data":"d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36"}
Oct 30 21:07:18.526222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:18.525803    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-58bzk" event={"ID":"3cd3ef1e-da27-4bed-b37f-addadd05af18","Type":"ContainerDied","Data":"99e7a3b43cf256424b5e5a714a8eeb82648d3e6f2b9bab59dc0c558205ca323b"}
Oct 30 21:07:18.526222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:18.525826    4918 scope.go:122] "RemoveContainer" containerID="d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36"
Oct 30 21:07:18.534161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:18.534144    4918 scope.go:122] "RemoveContainer" containerID="d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36"
Oct 30 21:07:18.534419 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:07:18.534376    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36\": container with ID starting with d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36 not found: ID does not exist" containerID="d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36"
Oct 30 21:07:18.534488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:18.534428    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36"} err="failed to get container status \"d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36\": rpc error: code = NotFound desc = could not find container \"d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36\": container with ID starting with d26b2cdd8423609661c375b5f322341055716c6b65df378c7e783981c0d09a36 not found: ID does not exist"
Oct 30 21:07:18.546657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:18.546635    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-58bzk"]
Oct 30 21:07:18.548680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:18.548662    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-58bzk"]
Oct 30 21:07:19.075369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:19.075341    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd3ef1e-da27-4bed-b37f-addadd05af18" path="/var/lib/kubelet/pods/3cd3ef1e-da27-4bed-b37f-addadd05af18/volumes"
Oct 30 21:07:28.531617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:28.531589    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6dbd7d59bd-wgg2g"
Oct 30 21:07:29.158150 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:07:29.158111    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:07:49.373020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.372986    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"]
Oct 30 21:07:49.373633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.373548    4918 cpu_manager.go:414] "containerMap: removing container" podUID="3cd3ef1e-da27-4bed-b37f-addadd05af18" containerName="authorino"
Oct 30 21:07:49.373633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.373567    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd3ef1e-da27-4bed-b37f-addadd05af18" containerName="authorino"
Oct 30 21:07:49.373763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.373673    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd3ef1e-da27-4bed-b37f-addadd05af18" containerName="authorino"
Oct 30 21:07:49.380491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.380470    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.389467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.389437    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"]
Oct 30 21:07:49.506366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.506328    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.506543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.506373    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.506543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.506507    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.506618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.506550    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.506656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.506616    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9n4h\" (UniqueName: \"kubernetes.io/projected/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-kube-api-access-p9n4h\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.506689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.506663    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.506721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.506700    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.607944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.607912    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.608062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.607956    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.608103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.608084    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.608146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.608120    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.608192 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.608158    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.608243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.608192    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.608302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.608245    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9n4h\" (UniqueName: \"kubernetes.io/projected/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-kube-api-access-p9n4h\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.609039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.608940    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.610849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.610823    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.611062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.611045    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.611161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.611041    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.611228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.611202    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.618118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.618096    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9n4h\" (UniqueName: \"kubernetes.io/projected/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-kube-api-access-p9n4h\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.618312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.618290    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fb99t\" (UID: \"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.693453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.693364    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:49.970742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.970541    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"]
Oct 30 21:07:49.973662 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:07:49.973628    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3c18ce_fc48_4be6_924c_5f4c89a8c8c1.slice/crio-8f7173f795b336c556821db085480fb45d9823d2fadbbd154446292544567233 WatchSource:0}: Error finding container 8f7173f795b336c556821db085480fb45d9823d2fadbbd154446292544567233: Status 404 returned error can't find the container with id 8f7173f795b336c556821db085480fb45d9823d2fadbbd154446292544567233
Oct 30 21:07:49.976251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.976222    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:07:49.976316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:49.976282    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:07:50.656511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:50.656473    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t" event={"ID":"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1","Type":"ContainerStarted","Data":"084fd7ad6c4df0e50c7c7930377776a7db232fd8d850d571ea281dc3bfb151a5"}
Oct 30 21:07:50.656898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:50.656530    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t" event={"ID":"9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1","Type":"ContainerStarted","Data":"8f7173f795b336c556821db085480fb45d9823d2fadbbd154446292544567233"}
Oct 30 21:07:50.656898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:50.656626    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:50.681970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:50.681923    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t" podStartSLOduration=1.6819103869999998 podStartE2EDuration="1.681910387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:50.679933757 +0000 UTC m=+383.726492981" watchObservedRunningTime="2026-10-30 21:07:50.681910387 +0000 UTC m=+383.728469531"
Oct 30 21:07:51.662226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:51.662195    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fb99t"
Oct 30 21:07:51.718675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:51.718646    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"]
Oct 30 21:07:51.718852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:51.718832    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t" podUID="a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" containerName="discovery" containerID="cri-o://28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a" gracePeriod=30
Oct 30 21:07:51.966182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:51.966145    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:07:52.132988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.132962    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-kubeconfig\") pod \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") "
Oct 30 21:07:52.132988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.132999    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-cacerts\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-cacerts\") pod \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") "
Oct 30 21:07:52.133179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.133035    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-local-certs\") pod \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") "
Oct 30 21:07:52.133179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.133063    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-kube-api-access-sd6c9\" (UniqueName: \"kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-kube-api-access-sd6c9\") pod \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") "
Oct 30 21:07:52.133261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.133237    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-ca-configmap\") pod \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") "
Oct 30 21:07:52.133312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.133290    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-token\" (UniqueName: \"kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-token\") pod \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") "
Oct 30 21:07:52.133415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.133373    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-dns-cert\") pod \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\" (UID: \"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0\") "
Oct 30 21:07:52.133654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.133629    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-ca-configmap" pod "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" (UID: "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:07:52.136228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.136196    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-kubeconfig" pod "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" (UID: "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:52.136228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.136207    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-token" pod "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" (UID: "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:52.136385 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.136233    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-cacerts" pod "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" (UID: "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:52.136385 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.136267    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-local-certs" pod "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" (UID: "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:07:52.136385 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.136299    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-kube-api-access-sd6c9" pod "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" (UID: "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0"). InnerVolumeSpecName "kube-api-access-sd6c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:07:52.136385 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.136313    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-dns-cert" pod "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" (UID: "a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:07:52.234810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.234776    4918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-dns-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:52.234810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.234804    4918 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-kubeconfig\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:52.234810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.234813    4918 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-cacerts\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:52.235039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.234822    4918 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-local-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:52.235039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.234834    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd6c9\" (UniqueName: \"kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-kube-api-access-sd6c9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:52.235039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.234843    4918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-csr-ca-configmap\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:52.235039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.234852    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:07:52.666276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.666198    4918 generic.go:356] "Generic (PLEG): container finished" podID="a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" containerID="28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a" exitCode=0
Oct 30 21:07:52.666276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.666254    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"
Oct 30 21:07:52.666749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.666290    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t" event={"ID":"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0","Type":"ContainerDied","Data":"28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a"}
Oct 30 21:07:52.666749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.666342    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t" event={"ID":"a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0","Type":"ContainerDied","Data":"e7462d7006bb0eedf67e67222da10dd6e16bcb4d6c81c086098b3fee664e473c"}
Oct 30 21:07:52.666749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.666363    4918 scope.go:122] "RemoveContainer" containerID="28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a"
Oct 30 21:07:52.675216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.675199    4918 scope.go:122] "RemoveContainer" containerID="28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a"
Oct 30 21:07:52.675476 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:07:52.675458    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a\": container with ID starting with 28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a not found: ID does not exist" containerID="28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a"
Oct 30 21:07:52.675531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.675486    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a"} err="failed to get container status \"28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a\": rpc error: code = NotFound desc = could not find container \"28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a\": container with ID starting with 28266d5e9a2d4b81d164ec042daed0db7dbab689be58bf0700997d25de09c43a not found: ID does not exist"
Oct 30 21:07:52.687860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.687841    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"]
Oct 30 21:07:52.691644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:52.691617    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-vxp5t"]
Oct 30 21:07:53.074632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:07:53.074600    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" path="/var/lib/kubelet/pods/a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0/volumes"
Oct 30 21:07:59.158271 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:07:59.158237    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:08:29.158690 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:08:29.158650    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:08:44.222266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.222227    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-g5b5m"]
Oct 30 21:08:44.222777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.222689    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" containerName="discovery"
Oct 30 21:08:44.222777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.222707    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" containerName="discovery"
Oct 30 21:08:44.222890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.222832    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f2f8cb-44c0-40b7-959a-4ebbf3d743c0" containerName="discovery"
Oct 30 21:08:44.225840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.225821    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-g5b5m"
Oct 30 21:08:44.239798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.239773    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zlhck\""
Oct 30 21:08:44.240208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.240189    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-g5b5m"]
Oct 30 21:08:44.252373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.252354    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:08:44.252482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.252447    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:08:44.310481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.310456    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-g5b5m"]
Oct 30 21:08:44.310641 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:08:44.310624    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6phwq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-g5b5m" podUID="4ce0ab61-5d09-47f7-af64-ea9822a2c129"
Oct 30 21:08:44.338655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.338630    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6phwq\" (UniqueName: \"kubernetes.io/projected/4ce0ab61-5d09-47f7-af64-ea9822a2c129-kube-api-access-6phwq\") pod \"authorino-6f6944bcdd-g5b5m\" (UID: \"4ce0ab61-5d09-47f7-af64-ea9822a2c129\") " pod="kuadrant-system/authorino-6f6944bcdd-g5b5m"
Oct 30 21:08:44.439694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.439660    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6phwq\" (UniqueName: \"kubernetes.io/projected/4ce0ab61-5d09-47f7-af64-ea9822a2c129-kube-api-access-6phwq\") pod \"authorino-6f6944bcdd-g5b5m\" (UID: \"4ce0ab61-5d09-47f7-af64-ea9822a2c129\") " pod="kuadrant-system/authorino-6f6944bcdd-g5b5m"
Oct 30 21:08:44.448176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.448145    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6phwq\" (UniqueName: \"kubernetes.io/projected/4ce0ab61-5d09-47f7-af64-ea9822a2c129-kube-api-access-6phwq\") pod \"authorino-6f6944bcdd-g5b5m\" (UID: \"4ce0ab61-5d09-47f7-af64-ea9822a2c129\") " pod="kuadrant-system/authorino-6f6944bcdd-g5b5m"
Oct 30 21:08:44.868844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.868811    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-g5b5m"
Oct 30 21:08:44.875908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.875887    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-g5b5m"
Oct 30 21:08:44.943891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.943859    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ce0ab61-5d09-47f7-af64-ea9822a2c129-kube-api-access-6phwq\" (UniqueName: \"kubernetes.io/projected/4ce0ab61-5d09-47f7-af64-ea9822a2c129-kube-api-access-6phwq\") pod \"4ce0ab61-5d09-47f7-af64-ea9822a2c129\" (UID: \"4ce0ab61-5d09-47f7-af64-ea9822a2c129\") "
Oct 30 21:08:44.946091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:44.946069    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ce0ab61-5d09-47f7-af64-ea9822a2c129-kube-api-access-6phwq" pod "4ce0ab61-5d09-47f7-af64-ea9822a2c129" (UID: "4ce0ab61-5d09-47f7-af64-ea9822a2c129"). InnerVolumeSpecName "kube-api-access-6phwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:08:45.044436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:45.044383    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6phwq\" (UniqueName: \"kubernetes.io/projected/4ce0ab61-5d09-47f7-af64-ea9822a2c129-kube-api-access-6phwq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:08:45.873233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:45.873195    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-g5b5m"
Oct 30 21:08:45.902968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:45.902934    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-g5b5m"]
Oct 30 21:08:45.908351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:45.908324    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-g5b5m"]
Oct 30 21:08:47.074622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:08:47.074594    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ce0ab61-5d09-47f7-af64-ea9822a2c129" path="/var/lib/kubelet/pods/4ce0ab61-5d09-47f7-af64-ea9822a2c129/volumes"
Oct 30 21:08:59.161109 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:08:59.161068    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:09:29.159257 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:09:29.159223    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:09:54.697802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.697700    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"]
Oct 30 21:09:54.700953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.700938    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.709358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.709326    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 21:09:54.710847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.710832    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 21:09:54.711086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.711071    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 21:09:54.713437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.713416    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vk6lq\""
Oct 30 21:09:54.715963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.715935    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"]
Oct 30 21:09:54.790712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.790680    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2825e8f7-cceb-46f1-87c9-23d30491e8b3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.790712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.790714    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr26c\" (UniqueName: \"kubernetes.io/projected/2825e8f7-cceb-46f1-87c9-23d30491e8b3-kube-api-access-nr26c\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.790902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.790745    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2825e8f7-cceb-46f1-87c9-23d30491e8b3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.790902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.790795    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2825e8f7-cceb-46f1-87c9-23d30491e8b3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.790902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.790863    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2825e8f7-cceb-46f1-87c9-23d30491e8b3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.790993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.790928    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2825e8f7-cceb-46f1-87c9-23d30491e8b3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.891866 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.891825    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2825e8f7-cceb-46f1-87c9-23d30491e8b3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.891866 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.891864    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2825e8f7-cceb-46f1-87c9-23d30491e8b3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.892081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.891882    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2825e8f7-cceb-46f1-87c9-23d30491e8b3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.892081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.891908    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2825e8f7-cceb-46f1-87c9-23d30491e8b3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.892081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.891959    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2825e8f7-cceb-46f1-87c9-23d30491e8b3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.892081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.891977    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr26c\" (UniqueName: \"kubernetes.io/projected/2825e8f7-cceb-46f1-87c9-23d30491e8b3-kube-api-access-nr26c\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.892340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.892313    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2825e8f7-cceb-46f1-87c9-23d30491e8b3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.892384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.892337    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2825e8f7-cceb-46f1-87c9-23d30491e8b3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.892384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.892362    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2825e8f7-cceb-46f1-87c9-23d30491e8b3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.894442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.894423    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2825e8f7-cceb-46f1-87c9-23d30491e8b3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.894685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.894667    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2825e8f7-cceb-46f1-87c9-23d30491e8b3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:54.900692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:54.900670    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr26c\" (UniqueName: \"kubernetes.io/projected/2825e8f7-cceb-46f1-87c9-23d30491e8b3-kube-api-access-nr26c\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x\" (UID: \"2825e8f7-cceb-46f1-87c9-23d30491e8b3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:55.013549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:55.013521    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:55.303946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:55.303877    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"]
Oct 30 21:09:55.307051 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:09:55.307017    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2825e8f7_cceb_46f1_87c9_23d30491e8b3.slice/crio-7e445139d54be0b60192886b57481ff5ab38c26bac480ca174b881ad6a806ef1 WatchSource:0}: Error finding container 7e445139d54be0b60192886b57481ff5ab38c26bac480ca174b881ad6a806ef1: Status 404 returned error can't find the container with id 7e445139d54be0b60192886b57481ff5ab38c26bac480ca174b881ad6a806ef1
Oct 30 21:09:56.150513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:56.150430    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x" event={"ID":"2825e8f7-cceb-46f1-87c9-23d30491e8b3","Type":"ContainerStarted","Data":"7e445139d54be0b60192886b57481ff5ab38c26bac480ca174b881ad6a806ef1"}
Oct 30 21:09:57.155868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:57.155837    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x" event={"ID":"2825e8f7-cceb-46f1-87c9-23d30491e8b3","Type":"ContainerStarted","Data":"1fd45c0f87ed62dfa295cf6f5cd499771f95f1581e4d1cf81caef9cf37547d15"}
Oct 30 21:09:58.013815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:09:58.013781    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:09:59.158141 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:09:59.158101    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:10:00.360737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.360680    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x" podStartSLOduration=5.167836766 podStartE2EDuration="6.360667759s" totalImagesPullingTime="1.192830993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:57.175846814 +0000 UTC m=+510.222405972" watchObservedRunningTime="2026-10-30 21:10:00.360667759 +0000 UTC m=+513.407226897"
Oct 30 21:10:00.361711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.361690    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"]
Oct 30 21:10:00.366135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.366118    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.377241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.377224    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 21:10:00.378341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.378321    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"]
Oct 30 21:10:00.444639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.444604    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b55befd-378d-4d0c-9d54-d9f500c08c22-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.444639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.444645    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b55befd-378d-4d0c-9d54-d9f500c08c22-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.444839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.444760    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b55befd-378d-4d0c-9d54-d9f500c08c22-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.444839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.444788    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b55befd-378d-4d0c-9d54-d9f500c08c22-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.444839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.444809    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4szh\" (UniqueName: \"kubernetes.io/projected/4b55befd-378d-4d0c-9d54-d9f500c08c22-kube-api-access-r4szh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.444981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.444865    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b55befd-378d-4d0c-9d54-d9f500c08c22-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.545509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.545477    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b55befd-378d-4d0c-9d54-d9f500c08c22-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.545509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.545512    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4szh\" (UniqueName: \"kubernetes.io/projected/4b55befd-378d-4d0c-9d54-d9f500c08c22-kube-api-access-r4szh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.545733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.545531    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b55befd-378d-4d0c-9d54-d9f500c08c22-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.545733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.545549    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b55befd-378d-4d0c-9d54-d9f500c08c22-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.545733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.545567    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b55befd-378d-4d0c-9d54-d9f500c08c22-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.545733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.545628    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b55befd-378d-4d0c-9d54-d9f500c08c22-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.545972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.545946    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b55befd-378d-4d0c-9d54-d9f500c08c22-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.546035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.545993    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b55befd-378d-4d0c-9d54-d9f500c08c22-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.546035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.546027    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b55befd-378d-4d0c-9d54-d9f500c08c22-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.548000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.547984    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b55befd-378d-4d0c-9d54-d9f500c08c22-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.548253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.548234    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b55befd-378d-4d0c-9d54-d9f500c08c22-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.554673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.554653    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4szh\" (UniqueName: \"kubernetes.io/projected/4b55befd-378d-4d0c-9d54-d9f500c08c22-kube-api-access-r4szh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-hmt8h\" (UID: \"4b55befd-378d-4d0c-9d54-d9f500c08c22\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.678839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.678767    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:00.951994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:00.951929    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"]
Oct 30 21:10:00.954788 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:10:00.954765    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b55befd_378d_4d0c_9d54_d9f500c08c22.slice/crio-a160d3408eaab8679488aa06efbfe6eaef488302c343bab18c1661c4fca00f8c WatchSource:0}: Error finding container a160d3408eaab8679488aa06efbfe6eaef488302c343bab18c1661c4fca00f8c: Status 404 returned error can't find the container with id a160d3408eaab8679488aa06efbfe6eaef488302c343bab18c1661c4fca00f8c
Oct 30 21:10:01.172034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:01.172002    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h" event={"ID":"4b55befd-378d-4d0c-9d54-d9f500c08c22","Type":"ContainerStarted","Data":"a160d3408eaab8679488aa06efbfe6eaef488302c343bab18c1661c4fca00f8c"}
Oct 30 21:10:02.177127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:02.177092    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h" event={"ID":"4b55befd-378d-4d0c-9d54-d9f500c08c22","Type":"ContainerStarted","Data":"46de56d58d65159ab76d489b89cf68ce881ba6265e1a18319cbedde6cc6dd868"}
Oct 30 21:10:02.196108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:02.196066    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h" podStartSLOduration=1.916980976 podStartE2EDuration="2.196052411s" totalImagesPullingTime="279.071435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:02.194495171 +0000 UTC m=+515.241054333" watchObservedRunningTime="2026-10-30 21:10:02.196052411 +0000 UTC m=+515.242611547"
Oct 30 21:10:02.679474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:02.679441    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:05.014579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:05.014533    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:10:05.027342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:05.027312    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:10:05.200282 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:05.200252    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x"
Oct 30 21:10:06.086856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.086827    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"]
Oct 30 21:10:06.090025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.090005    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.093974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.093949    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 21:10:06.099658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.099635    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"]
Oct 30 21:10:06.195298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.195266    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a3d45f-aa19-4b00-b461-2896fe5929f6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.195298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.195303    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a3d45f-aa19-4b00-b461-2896fe5929f6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.195605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.195323    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a3d45f-aa19-4b00-b461-2896fe5929f6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.195605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.195413    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a3d45f-aa19-4b00-b461-2896fe5929f6-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.195605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.195450    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a3d45f-aa19-4b00-b461-2896fe5929f6-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.195605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.195528    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4w5p\" (UniqueName: \"kubernetes.io/projected/83a3d45f-aa19-4b00-b461-2896fe5929f6-kube-api-access-v4w5p\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.296142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.296098    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a3d45f-aa19-4b00-b461-2896fe5929f6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.296321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.296147    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a3d45f-aa19-4b00-b461-2896fe5929f6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.296321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.296270    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a3d45f-aa19-4b00-b461-2896fe5929f6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.296490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.296322    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a3d45f-aa19-4b00-b461-2896fe5929f6-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.296490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.296371    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a3d45f-aa19-4b00-b461-2896fe5929f6-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.296490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.296423    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4w5p\" (UniqueName: \"kubernetes.io/projected/83a3d45f-aa19-4b00-b461-2896fe5929f6-kube-api-access-v4w5p\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.296652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.296636    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a3d45f-aa19-4b00-b461-2896fe5929f6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.296741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.296716    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a3d45f-aa19-4b00-b461-2896fe5929f6-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.296741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.296736    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a3d45f-aa19-4b00-b461-2896fe5929f6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.298848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.298827    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a3d45f-aa19-4b00-b461-2896fe5929f6-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.298977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.298882    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a3d45f-aa19-4b00-b461-2896fe5929f6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.304261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.304241    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4w5p\" (UniqueName: \"kubernetes.io/projected/83a3d45f-aa19-4b00-b461-2896fe5929f6-kube-api-access-v4w5p\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf\" (UID: \"83a3d45f-aa19-4b00-b461-2896fe5929f6\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.408684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.408614    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:06.677712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:06.677635    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"]
Oct 30 21:10:06.680930 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:10:06.680896    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a3d45f_aa19_4b00_b461_2896fe5929f6.slice/crio-6efb77b18079bd7a29ab101523c27993bf684991d31e118a2da579ee501a9eba WatchSource:0}: Error finding container 6efb77b18079bd7a29ab101523c27993bf684991d31e118a2da579ee501a9eba: Status 404 returned error can't find the container with id 6efb77b18079bd7a29ab101523c27993bf684991d31e118a2da579ee501a9eba
Oct 30 21:10:07.199878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.199845    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf" event={"ID":"83a3d45f-aa19-4b00-b461-2896fe5929f6","Type":"ContainerStarted","Data":"6317930c1e99d6c03a2f085789b64213e19a482826d0f2a2d4603f51b6894be8"}
Oct 30 21:10:07.200290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.199895    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf" event={"ID":"83a3d45f-aa19-4b00-b461-2896fe5929f6","Type":"ContainerStarted","Data":"6efb77b18079bd7a29ab101523c27993bf684991d31e118a2da579ee501a9eba"}
Oct 30 21:10:07.221882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.221834    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf" podStartSLOduration=0.927467245 podStartE2EDuration="1.221819308s" totalImagesPullingTime="294.352063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:07.220911774 +0000 UTC m=+520.267470938" watchObservedRunningTime="2026-10-30 21:10:07.221819308 +0000 UTC m=+520.268378459"
Oct 30 21:10:07.409061 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.408984    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:07.636893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.636869    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"]
Oct 30 21:10:07.640288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.640272    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.648838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.648817    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 21:10:07.656368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.656350    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"]
Oct 30 21:10:07.810124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.810085    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ec8e779-955f-4570-81ad-480d8cefca7a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.810124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.810122    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec8e779-955f-4570-81ad-480d8cefca7a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.810343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.810154    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ec8e779-955f-4570-81ad-480d8cefca7a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.810343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.810289    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkg54\" (UniqueName: \"kubernetes.io/projected/1ec8e779-955f-4570-81ad-480d8cefca7a-kube-api-access-wkg54\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.810343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.810320    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ec8e779-955f-4570-81ad-480d8cefca7a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.810499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.810345    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ec8e779-955f-4570-81ad-480d8cefca7a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.911125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.911088    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkg54\" (UniqueName: \"kubernetes.io/projected/1ec8e779-955f-4570-81ad-480d8cefca7a-kube-api-access-wkg54\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.911303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.911131    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ec8e779-955f-4570-81ad-480d8cefca7a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.911303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.911163    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ec8e779-955f-4570-81ad-480d8cefca7a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.911303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.911205    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ec8e779-955f-4570-81ad-480d8cefca7a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.911303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.911232    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec8e779-955f-4570-81ad-480d8cefca7a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.911303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.911278    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ec8e779-955f-4570-81ad-480d8cefca7a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.911649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.911625    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ec8e779-955f-4570-81ad-480d8cefca7a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.911714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.911631    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ec8e779-955f-4570-81ad-480d8cefca7a-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.911714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.911673    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ec8e779-955f-4570-81ad-480d8cefca7a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.913549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.913530    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ec8e779-955f-4570-81ad-480d8cefca7a-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.913841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.913823    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ec8e779-955f-4570-81ad-480d8cefca7a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.922694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.922674    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkg54\" (UniqueName: \"kubernetes.io/projected/1ec8e779-955f-4570-81ad-480d8cefca7a-kube-api-access-wkg54\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq\" (UID: \"1ec8e779-955f-4570-81ad-480d8cefca7a\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:07.963618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:07.963593    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:08.236630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:08.236605    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"]
Oct 30 21:10:08.239318 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:10:08.239294    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ec8e779_955f_4570_81ad_480d8cefca7a.slice/crio-8898934c731826c9d2efd23b3ea1a73c3452026f73463275a49c9aebc211cf8d WatchSource:0}: Error finding container 8898934c731826c9d2efd23b3ea1a73c3452026f73463275a49c9aebc211cf8d: Status 404 returned error can't find the container with id 8898934c731826c9d2efd23b3ea1a73c3452026f73463275a49c9aebc211cf8d
Oct 30 21:10:09.208774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:09.208731    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" event={"ID":"1ec8e779-955f-4570-81ad-480d8cefca7a","Type":"ContainerStarted","Data":"f57069970ee61f394e80cc6456a9eae892a390e2080151be3c918bce5f09de36"}
Oct 30 21:10:09.208965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:09.208785    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" event={"ID":"1ec8e779-955f-4570-81ad-480d8cefca7a","Type":"ContainerStarted","Data":"8898934c731826c9d2efd23b3ea1a73c3452026f73463275a49c9aebc211cf8d"}
Oct 30 21:10:09.231170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:09.231128    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podStartSLOduration=1.9447501950000001 podStartE2EDuration="2.231115363s" totalImagesPullingTime="286.365168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:09.230950344 +0000 UTC m=+522.277509491" watchObservedRunningTime="2026-10-30 21:10:09.231115363 +0000 UTC m=+522.277674508"
Oct 30 21:10:09.964273 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:09.964229    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:10.679197 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:10.679165    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:10.691673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:10.691647    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:11.228782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:11.228751    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-hmt8h"
Oct 30 21:10:16.409581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:16.409539    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:16.421948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:16.421917    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:17.253315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:17.253279    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf"
Oct 30 21:10:17.964476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:17.964445    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:17.977010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:17.976989    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:18.257504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:18.257479    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:10:28.721029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.720992    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"]
Oct 30 21:10:28.727863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.727840    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.736815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.736800    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-l56fx\""
Oct 30 21:10:28.741889 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.741868    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"]
Oct 30 21:10:28.779548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.779524    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.779674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.779554    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0439b0d-ff5f-49ab-87c3-afada363adcb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.779674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.779570    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgtqr\" (UniqueName: \"kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-kube-api-access-fgtqr\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.779739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.779683    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.779739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.779716    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.779798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.779742    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.779798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.779768    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.779861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.779839    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.779927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.779905    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.880606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.880572    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.880606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.880607    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.880818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.880635    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.880818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.880666    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.880818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.880703    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.880818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.880729    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0439b0d-ff5f-49ab-87c3-afada363adcb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.880818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.880752    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgtqr\" (UniqueName: \"kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-kube-api-access-fgtqr\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.880818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.880815    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.881108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.880854    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.881108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.881038    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.881214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.881147    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.881214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.881166    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.881214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.881205    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.881451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.881433    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0439b0d-ff5f-49ab-87c3-afada363adcb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.883121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.883096    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.883233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.883134    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.890850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.890822    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:28.891093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:28.891077    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgtqr\" (UniqueName: \"kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-kube-api-access-fgtqr\") pod \"maas-default-gateway-openshift-default-75bf867b6c-smshz\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:29.042312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:29.042222    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:29.159601 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:10:29.159568    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:10:29.323221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:29.323195    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"]
Oct 30 21:10:30.300323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:30.300292    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" event={"ID":"a0439b0d-ff5f-49ab-87c3-afada363adcb","Type":"ContainerStarted","Data":"070351413d354aa5c89096fd0b037853bd98e9d2835ce075dc3b0979305f2be1"}
Oct 30 21:10:32.302763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:32.302737    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:32.303031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:32.302799    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:32.303031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:32.302829    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:33.315820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:33.315782    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" event={"ID":"a0439b0d-ff5f-49ab-87c3-afada363adcb","Type":"ContainerStarted","Data":"a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd"}
Oct 30 21:10:33.337407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:33.337344    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" podStartSLOduration=2.364465258 podStartE2EDuration="5.337327952s" totalImagesPullingTime="2.972862694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:33.336268265 +0000 UTC m=+546.382827435" watchObservedRunningTime="2026-10-30 21:10:33.337327952 +0000 UTC m=+546.383887097"
Oct 30 21:10:34.042582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:34.042547    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:34.044200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:34.044174    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-smshz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:10:34.044312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:34.044232    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" podUID="a0439b0d-ff5f-49ab-87c3-afada363adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 30 21:10:35.042798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:35.042771    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-smshz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:10:35.043198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:35.042819    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" podUID="a0439b0d-ff5f-49ab-87c3-afada363adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 30 21:10:36.042697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:36.042665    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-smshz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:10:36.042958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:36.042728    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" podUID="a0439b0d-ff5f-49ab-87c3-afada363adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 30 21:10:37.043282 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:37.043246    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-smshz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:10:37.043719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:37.043326    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" podUID="a0439b0d-ff5f-49ab-87c3-afada363adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 30 21:10:38.042860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:38.042821    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-smshz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:10:38.043044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:38.042875    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" podUID="a0439b0d-ff5f-49ab-87c3-afada363adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 30 21:10:39.043122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:39.043079    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-smshz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:10:39.043625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:39.043137    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" podUID="a0439b0d-ff5f-49ab-87c3-afada363adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 30 21:10:40.470298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:40.470268    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:40.470779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:40.470674    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:40.471542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:40.471526    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:10:43.749210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.749176    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"]
Oct 30 21:10:43.752847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.752832    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.769130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.769106    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"]
Oct 30 21:10:43.796225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.796198    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"]
Oct 30 21:10:43.800065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.800046    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:43.805178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.805149    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.805309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.805211    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.805309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.805259    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.805424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.805378    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5z89\" (UniqueName: \"kubernetes.io/projected/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-kube-api-access-j5z89\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.805504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.805473    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.805558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.805522    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.805603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.805568    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.805684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.805667    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.805734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.805693    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.813702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.813677    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"]
Oct 30 21:10:43.906461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.906431    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:43.906628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.906471    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.906628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.906490    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:43.906628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.906565    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:43.906732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.906663    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98bt2\" (UniqueName: \"kubernetes.io/projected/40508368-7d69-41e3-a2ee-0be190a20281-kube-api-access-98bt2\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:43.906765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.906736    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.906801 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.906771    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.906801 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.906796    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:43.906890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.906816    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.906890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.906861    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.906974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.906921    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40508368-7d69-41e3-a2ee-0be190a20281-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:43.907024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.906977    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:43.907024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.907012    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.907167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.907143    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40508368-7d69-41e3-a2ee-0be190a20281-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:43.907229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.907187    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.907229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.907148    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.907337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.907248    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5z89\" (UniqueName: \"kubernetes.io/projected/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-kube-api-access-j5z89\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.907337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.907259    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.907337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.907273    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.907337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.907272    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.907337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.907305    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40508368-7d69-41e3-a2ee-0be190a20281-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:43.907627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.907501    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.907627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.907552    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.909644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.909623    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.909751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.909699    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.915001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.914979    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:43.915228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:43.915210    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5z89\" (UniqueName: \"kubernetes.io/projected/de5f2249-cc22-496c-b84a-8d1fcf7c5fc7-kube-api-access-j5z89\") pod \"maas-default-gateway-openshift-default-75bf867b6c-w9kjp\" (UID: \"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:44.008634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.008550    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.008634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.008610    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40508368-7d69-41e3-a2ee-0be190a20281-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.008849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.008657    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40508368-7d69-41e3-a2ee-0be190a20281-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.008849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.008683    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.008849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.008721    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.008849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.008746    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.008849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.008770    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bt2\" (UniqueName: \"kubernetes.io/projected/40508368-7d69-41e3-a2ee-0be190a20281-kube-api-access-98bt2\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.009102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.008870    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.009102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.008970    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40508368-7d69-41e3-a2ee-0be190a20281-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.009102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.008973    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.009102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.009046    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.009242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.009223    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.009296 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.009276    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.009461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.009439    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40508368-7d69-41e3-a2ee-0be190a20281-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.011269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.011248    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40508368-7d69-41e3-a2ee-0be190a20281-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.011443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.011429    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40508368-7d69-41e3-a2ee-0be190a20281-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.017428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.017375    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40508368-7d69-41e3-a2ee-0be190a20281-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.017657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.017641    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98bt2\" (UniqueName: \"kubernetes.io/projected/40508368-7d69-41e3-a2ee-0be190a20281-kube-api-access-98bt2\") pod \"maas-default-gateway-openshift-default-75bf867b6c-d54n7\" (UID: \"40508368-7d69-41e3-a2ee-0be190a20281\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.068872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.068842    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:44.113015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.112979    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:44.351841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.351817    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"]
Oct 30 21:10:44.353909 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:10:44.353877    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5f2249_cc22_496c_b84a_8d1fcf7c5fc7.slice/crio-3c31480335dbb5bb591599f274fc3477e23e7735cceadbf2dae2d2221e410fdb WatchSource:0}: Error finding container 3c31480335dbb5bb591599f274fc3477e23e7735cceadbf2dae2d2221e410fdb: Status 404 returned error can't find the container with id 3c31480335dbb5bb591599f274fc3477e23e7735cceadbf2dae2d2221e410fdb
Oct 30 21:10:44.357117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.357084    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:44.357186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.357155    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:44.357222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.357184    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:44.363240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.363211    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp" event={"ID":"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7","Type":"ContainerStarted","Data":"3c31480335dbb5bb591599f274fc3477e23e7735cceadbf2dae2d2221e410fdb"}
Oct 30 21:10:44.406218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.406193    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"]
Oct 30 21:10:44.407969 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:10:44.407945    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40508368_7d69_41e3_a2ee_0be190a20281.slice/crio-b203aa84dde36d506c0f439507e77c3f6fbaaa8d2649e397b3f7cf3fbcc8e182 WatchSource:0}: Error finding container b203aa84dde36d506c0f439507e77c3f6fbaaa8d2649e397b3f7cf3fbcc8e182: Status 404 returned error can't find the container with id b203aa84dde36d506c0f439507e77c3f6fbaaa8d2649e397b3f7cf3fbcc8e182
Oct 30 21:10:44.411337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.411312    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:44.411445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.411372    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:44.411445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:44.411418    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:10:45.369443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:45.369407    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp" event={"ID":"de5f2249-cc22-496c-b84a-8d1fcf7c5fc7","Type":"ContainerStarted","Data":"22de8a7a3755325f5ac65ea7c8cbe294c762574116f469780ea6ffb631f21ef7"}
Oct 30 21:10:45.371262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:45.371232    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7" event={"ID":"40508368-7d69-41e3-a2ee-0be190a20281","Type":"ContainerStarted","Data":"e587145b50dda5541b33e09d8a7a8475f876c75e3fc89c3004de8e008c507349"}
Oct 30 21:10:45.371433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:45.371272    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7" event={"ID":"40508368-7d69-41e3-a2ee-0be190a20281","Type":"ContainerStarted","Data":"b203aa84dde36d506c0f439507e77c3f6fbaaa8d2649e397b3f7cf3fbcc8e182"}
Oct 30 21:10:45.406416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:45.395595    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp" podStartSLOduration=2.395574467 podStartE2EDuration="2.395574467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:45.393861521 +0000 UTC m=+558.440420687" watchObservedRunningTime="2026-10-30 21:10:45.395574467 +0000 UTC m=+558.442133614"
Oct 30 21:10:45.437838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:45.437773    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7" podStartSLOduration=2.437756317 podStartE2EDuration="2.437756317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:45.436154915 +0000 UTC m=+558.482714057" watchObservedRunningTime="2026-10-30 21:10:45.437756317 +0000 UTC m=+558.484315531"
Oct 30 21:10:46.069509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:46.069447    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:46.071062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:46.071032    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-w9kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:10:46.071205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:46.071095    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp" podUID="de5f2249-cc22-496c-b84a-8d1fcf7c5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Oct 30 21:10:46.115465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:46.115021    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:46.115465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:46.115144    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-d54n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:10:46.115465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:46.115185    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7" podUID="40508368-7d69-41e3-a2ee-0be190a20281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 30 21:10:47.071822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:47.071786    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-w9kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:10:47.072333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:47.071834    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp" podUID="de5f2249-cc22-496c-b84a-8d1fcf7c5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Oct 30 21:10:47.115974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:47.115939    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-d54n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:10:47.116284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:47.116240    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7" podUID="40508368-7d69-41e3-a2ee-0be190a20281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 30 21:10:48.069525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:48.069483    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-w9kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:10:48.069740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:48.069556    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp" podUID="de5f2249-cc22-496c-b84a-8d1fcf7c5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Oct 30 21:10:48.115837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:48.115800    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-d54n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:10:48.116442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:48.116380    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7" podUID="40508368-7d69-41e3-a2ee-0be190a20281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 30 21:10:49.070750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:49.070714    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-w9kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:10:49.070943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:49.070791    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp" podUID="de5f2249-cc22-496c-b84a-8d1fcf7c5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Oct 30 21:10:49.121936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:49.121901    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-d54n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:10:49.122603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:49.122555    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7" podUID="40508368-7d69-41e3-a2ee-0be190a20281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 30 21:10:50.069989 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:50.069955    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-w9kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:10:50.070170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:50.070004    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp" podUID="de5f2249-cc22-496c-b84a-8d1fcf7c5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Oct 30 21:10:50.113822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:50.113785    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-d54n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:10:50.114102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:50.114074    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7" podUID="40508368-7d69-41e3-a2ee-0be190a20281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 30 21:10:51.069800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:51.069758    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-w9kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:10:51.070220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:51.069846    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp" podUID="de5f2249-cc22-496c-b84a-8d1fcf7c5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Oct 30 21:10:51.113714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:51.113683    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-d54n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 30 21:10:51.113884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:51.113736    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7" podUID="40508368-7d69-41e3-a2ee-0be190a20281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 30 21:10:52.614897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:52.614857    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:52.616104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:52.616083    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:52.617067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:52.617040    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp"
Oct 30 21:10:52.645817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:52.645785    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:52.646342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:52.646233    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:52.647244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:10:52.647222    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7"
Oct 30 21:10:59.158206 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:10:59.158164    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:11:05.201523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.201490    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"]
Oct 30 21:11:05.209305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.209285    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.220288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.220264    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"]
Oct 30 21:11:05.221967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.221943    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e2cef583-openshift-default-dockercfg-xmp4g\""
Oct 30 21:11:05.324914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.324859    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istiod-ca-cert\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.325092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.324938    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bprq\" (UniqueName: \"kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-kube-api-access-9bprq\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.325092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.325033    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-certs\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.325176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.325097    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-data\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.325219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.325174    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-envoy\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.325259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.325235    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-credential-socket\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.325310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.325291    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-podinfo\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.325346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.325328    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-socket\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.325401 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.325349    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-token\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.426420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.426363    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-envoy\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.426576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.426485    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-credential-socket\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.426576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.426510    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-podinfo\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.426576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.426536    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-socket\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.426576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.426564    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-token\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.426796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.426597    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istiod-ca-cert\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.426796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.426622    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bprq\" (UniqueName: \"kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-kube-api-access-9bprq\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.426796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.426653    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-certs\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.426796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.426715    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-data\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.427034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.426877    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-credential-socket\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.427034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.426922    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-socket\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.427165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.427106    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-data\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.427225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.427201    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-certs\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.427771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.427748    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istiod-ca-cert\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.428923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.428899    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-envoy\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.429140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.429123    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-podinfo\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.439751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.439727    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-token\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.439895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.439877    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bprq\" (UniqueName: \"kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-kube-api-access-9bprq\") pod \"e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") " pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.526661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.526631    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:05.848441 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:05.848369    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db5e191_2a8e_40d2_819c_e9c00f33f07e.slice/crio-3840a5d3d195e2ed32e7f3282cf61da7b019e27728c6ee79ad3385a60ffbcfc3 WatchSource:0}: Error finding container 3840a5d3d195e2ed32e7f3282cf61da7b019e27728c6ee79ad3385a60ffbcfc3: Status 404 returned error can't find the container with id 3840a5d3d195e2ed32e7f3282cf61da7b019e27728c6ee79ad3385a60ffbcfc3
Oct 30 21:11:05.850812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.850780    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:05.850912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.850849    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:05.850912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.850870    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"]
Oct 30 21:11:05.850995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:05.850877    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:06.482019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:06.481985    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" event={"ID":"1db5e191-2a8e-40d2-819c-e9c00f33f07e","Type":"ContainerStarted","Data":"7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9"}
Oct 30 21:11:06.482461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:06.482054    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" event={"ID":"1db5e191-2a8e-40d2-819c-e9c00f33f07e","Type":"ContainerStarted","Data":"3840a5d3d195e2ed32e7f3282cf61da7b019e27728c6ee79ad3385a60ffbcfc3"}
Oct 30 21:11:06.518960 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:06.518911    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" podStartSLOduration=1.518898603 podStartE2EDuration="1.518898603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:06.516658718 +0000 UTC m=+579.563217889" watchObservedRunningTime="2026-10-30 21:11:06.518898603 +0000 UTC m=+579.565457748"
Oct 30 21:11:06.527203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:06.527175    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:06.532115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:06.532093    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:06.989990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:06.989916    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"]
Oct 30 21:11:06.994316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:06.994295    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:06.999074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:06.999055    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-111fd691-openshift-default-dockercfg-95tk9\""
Oct 30 21:11:07.009154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.009133    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"]
Oct 30 21:11:07.164119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.160451    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-token\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.164119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.160500    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24d688c-575f-4c80-8745-8cc0d72ad30c-istiod-ca-cert\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.164119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.160550    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-socket\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.164119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.160606    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-podinfo\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.164119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.160653    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-credential-socket\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.164119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.160707    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-envoy\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.164119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.160759    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f5cf\" (UniqueName: \"kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-kube-api-access-9f5cf\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.164119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.160798    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-data\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.164119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.160836    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-certs\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.261457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.261368    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-podinfo\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.261457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.261432    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-credential-socket\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.261679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.261462    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-envoy\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.261679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.261493    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f5cf\" (UniqueName: \"kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-kube-api-access-9f5cf\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.261679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.261516    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-data\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.261679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.261536    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-certs\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.261679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.261575    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-token\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.261679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.261591    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24d688c-575f-4c80-8745-8cc0d72ad30c-istiod-ca-cert\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.261924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.261719    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-socket\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.262211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.262183    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-data\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.262211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.262207    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-socket\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.262457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.262280    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-credential-socket\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.262524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.262451    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-certs\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.262617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.262599    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24d688c-575f-4c80-8745-8cc0d72ad30c-istiod-ca-cert\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.264070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.264043    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-podinfo\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.264070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.264065    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-envoy\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.270006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.269981    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-token\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.270110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.270047    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f5cf\" (UniqueName: \"kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-kube-api-access-9f5cf\") pod \"e2e-ait-111fd691-openshift-default-598486d85f-njfwx\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") " pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.308318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.308292    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:07.495311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.495275    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:07.496176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.496152    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:07.587069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.587046    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"]
Oct 30 21:11:07.589160 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:07.589135    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24d688c_575f_4c80_8745_8cc0d72ad30c.slice/crio-f6fc00e5d746352482a506ba42e8335bb51fd7a78a783376de01a6b5f8333b4e WatchSource:0}: Error finding container f6fc00e5d746352482a506ba42e8335bb51fd7a78a783376de01a6b5f8333b4e: Status 404 returned error can't find the container with id f6fc00e5d746352482a506ba42e8335bb51fd7a78a783376de01a6b5f8333b4e
Oct 30 21:11:07.591792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.591759    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:07.591881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.591826    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:07.591881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:07.591854    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:08.501532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:08.501492    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" event={"ID":"f24d688c-575f-4c80-8745-8cc0d72ad30c","Type":"ContainerStarted","Data":"0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4"}
Oct 30 21:11:08.502010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:08.501551    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" event={"ID":"f24d688c-575f-4c80-8745-8cc0d72ad30c","Type":"ContainerStarted","Data":"f6fc00e5d746352482a506ba42e8335bb51fd7a78a783376de01a6b5f8333b4e"}
Oct 30 21:11:08.527602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:08.527560    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" podStartSLOduration=2.5275479709999997 podStartE2EDuration="2.527547971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:08.52606669 +0000 UTC m=+581.572625855" watchObservedRunningTime="2026-10-30 21:11:08.527547971 +0000 UTC m=+581.574107116"
Oct 30 21:11:09.309154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.309123    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:09.313898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.313872    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:09.506103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.506068    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:09.506920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.506904    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:09.610985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.610914    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"]
Oct 30 21:11:09.614431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.614412    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"
Oct 30 21:11:09.621919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.621890    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e2cef583\""
Oct 30 21:11:09.630627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.630603    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"]
Oct 30 21:11:09.642673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.642651    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e2cef583-dockercfg-lvqm4\""
Oct 30 21:11:09.656956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.656932    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"]
Oct 30 21:11:09.660422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.660405    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"
Oct 30 21:11:09.674705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.674682    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"]
Oct 30 21:11:09.796237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.796184    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58256\" (UniqueName: \"kubernetes.io/projected/2ea25b16-f21e-444d-a4fd-0faff56edf63-kube-api-access-58256\") pod \"payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts\" (UID: \"2ea25b16-f21e-444d-a4fd-0faff56edf63\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"
Oct 30 21:11:09.796465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.796346    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt624\" (UniqueName: \"kubernetes.io/projected/f10f3396-84b6-4fc8-ae81-94802cd42e09-kube-api-access-zt624\") pod \"payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg\" (UID: \"f10f3396-84b6-4fc8-ae81-94802cd42e09\") " pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"
Oct 30 21:11:09.796465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.796455    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea25b16-f21e-444d-a4fd-0faff56edf63-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts\" (UID: \"2ea25b16-f21e-444d-a4fd-0faff56edf63\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"
Oct 30 21:11:09.796591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.796518    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10f3396-84b6-4fc8-ae81-94802cd42e09-plugins-config-volume\") pod \"payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg\" (UID: \"f10f3396-84b6-4fc8-ae81-94802cd42e09\") " pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"
Oct 30 21:11:09.897713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.897640    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt624\" (UniqueName: \"kubernetes.io/projected/f10f3396-84b6-4fc8-ae81-94802cd42e09-kube-api-access-zt624\") pod \"payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg\" (UID: \"f10f3396-84b6-4fc8-ae81-94802cd42e09\") " pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"
Oct 30 21:11:09.897713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.897679    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea25b16-f21e-444d-a4fd-0faff56edf63-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts\" (UID: \"2ea25b16-f21e-444d-a4fd-0faff56edf63\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"
Oct 30 21:11:09.897713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.897714    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10f3396-84b6-4fc8-ae81-94802cd42e09-plugins-config-volume\") pod \"payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg\" (UID: \"f10f3396-84b6-4fc8-ae81-94802cd42e09\") " pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"
Oct 30 21:11:09.898018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.897765    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58256\" (UniqueName: \"kubernetes.io/projected/2ea25b16-f21e-444d-a4fd-0faff56edf63-kube-api-access-58256\") pod \"payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts\" (UID: \"2ea25b16-f21e-444d-a4fd-0faff56edf63\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"
Oct 30 21:11:09.898467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.898445    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea25b16-f21e-444d-a4fd-0faff56edf63-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts\" (UID: \"2ea25b16-f21e-444d-a4fd-0faff56edf63\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"
Oct 30 21:11:09.898562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.898495    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10f3396-84b6-4fc8-ae81-94802cd42e09-plugins-config-volume\") pod \"payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg\" (UID: \"f10f3396-84b6-4fc8-ae81-94802cd42e09\") " pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"
Oct 30 21:11:09.913069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.913027    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58256\" (UniqueName: \"kubernetes.io/projected/2ea25b16-f21e-444d-a4fd-0faff56edf63-kube-api-access-58256\") pod \"payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts\" (UID: \"2ea25b16-f21e-444d-a4fd-0faff56edf63\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"
Oct 30 21:11:09.921015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.920895    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt624\" (UniqueName: \"kubernetes.io/projected/f10f3396-84b6-4fc8-ae81-94802cd42e09-kube-api-access-zt624\") pod \"payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg\" (UID: \"f10f3396-84b6-4fc8-ae81-94802cd42e09\") " pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"
Oct 30 21:11:09.935776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.935747    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"
Oct 30 21:11:09.973035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:09.973009    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"
Oct 30 21:11:10.249440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.247791    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"]
Oct 30 21:11:10.283059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.283038    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"]
Oct 30 21:11:10.285095 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:10.285075    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10f3396_84b6_4fc8_ae81_94802cd42e09.slice/crio-abf6301b593d2c473da0f4df990e1a84647bcba924e2600c7cc463ff1edf1982 WatchSource:0}: Error finding container abf6301b593d2c473da0f4df990e1a84647bcba924e2600c7cc463ff1edf1982: Status 404 returned error can't find the container with id abf6301b593d2c473da0f4df990e1a84647bcba924e2600c7cc463ff1edf1982
Oct 30 21:11:10.511849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.511759    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts" event={"ID":"2ea25b16-f21e-444d-a4fd-0faff56edf63","Type":"ContainerStarted","Data":"f576c3a3197e011f3eb4a4e64d1088edef333c0c1c5b4566971601f086323951"}
Oct 30 21:11:10.512960 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.512936    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg" event={"ID":"f10f3396-84b6-4fc8-ae81-94802cd42e09","Type":"ContainerStarted","Data":"abf6301b593d2c473da0f4df990e1a84647bcba924e2600c7cc463ff1edf1982"}
Oct 30 21:11:10.585688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.585657    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"]
Oct 30 21:11:10.590219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.590204    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"
Oct 30 21:11:10.605031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.605007    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"]
Oct 30 21:11:10.656326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.656214    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6eedd82b-dockercfg-sps7p\""
Oct 30 21:11:10.656326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.656282    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6eedd82b\""
Oct 30 21:11:10.724879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.724850    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4bm5\" (UniqueName: \"kubernetes.io/projected/adfba4d2-eb18-484f-af0b-e4eb2771bb95-kube-api-access-s4bm5\") pod \"payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w\" (UID: \"adfba4d2-eb18-484f-af0b-e4eb2771bb95\") " pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"
Oct 30 21:11:10.725049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.724920    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfba4d2-eb18-484f-af0b-e4eb2771bb95-plugins-config-volume\") pod \"payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w\" (UID: \"adfba4d2-eb18-484f-af0b-e4eb2771bb95\") " pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"
Oct 30 21:11:10.826076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.825999    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4bm5\" (UniqueName: \"kubernetes.io/projected/adfba4d2-eb18-484f-af0b-e4eb2771bb95-kube-api-access-s4bm5\") pod \"payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w\" (UID: \"adfba4d2-eb18-484f-af0b-e4eb2771bb95\") " pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"
Oct 30 21:11:10.826242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.826121    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfba4d2-eb18-484f-af0b-e4eb2771bb95-plugins-config-volume\") pod \"payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w\" (UID: \"adfba4d2-eb18-484f-af0b-e4eb2771bb95\") " pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"
Oct 30 21:11:10.827007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.826987    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfba4d2-eb18-484f-af0b-e4eb2771bb95-plugins-config-volume\") pod \"payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w\" (UID: \"adfba4d2-eb18-484f-af0b-e4eb2771bb95\") " pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"
Oct 30 21:11:10.837483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.837446    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4bm5\" (UniqueName: \"kubernetes.io/projected/adfba4d2-eb18-484f-af0b-e4eb2771bb95-kube-api-access-s4bm5\") pod \"payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w\" (UID: \"adfba4d2-eb18-484f-af0b-e4eb2771bb95\") " pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"
Oct 30 21:11:10.903526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:10.903488    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"
Oct 30 21:11:11.256759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:11.256692    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"]
Oct 30 21:11:11.260017 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:11.259985    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfba4d2_eb18_484f_af0b_e4eb2771bb95.slice/crio-5f9b173007a39b1b62a84dec338b8dce344343e0793640f7bd2432b67b94bdc2 WatchSource:0}: Error finding container 5f9b173007a39b1b62a84dec338b8dce344343e0793640f7bd2432b67b94bdc2: Status 404 returned error can't find the container with id 5f9b173007a39b1b62a84dec338b8dce344343e0793640f7bd2432b67b94bdc2
Oct 30 21:11:11.521497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:11.521422    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w" event={"ID":"adfba4d2-eb18-484f-af0b-e4eb2771bb95","Type":"ContainerStarted","Data":"5f9b173007a39b1b62a84dec338b8dce344343e0793640f7bd2432b67b94bdc2"}
Oct 30 21:11:13.546902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.546866    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts" event={"ID":"2ea25b16-f21e-444d-a4fd-0faff56edf63","Type":"ContainerStarted","Data":"c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3"}
Oct 30 21:11:13.547580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.547533    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"
Oct 30 21:11:13.553502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.553470    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg" event={"ID":"f10f3396-84b6-4fc8-ae81-94802cd42e09","Type":"ContainerStarted","Data":"e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48"}
Oct 30 21:11:13.553686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.553653    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"
Oct 30 21:11:13.557239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.557167    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w" event={"ID":"adfba4d2-eb18-484f-af0b-e4eb2771bb95","Type":"ContainerStarted","Data":"7c3b15a6cab67ad84ad51d47bc44c9d41be43d7348ef2a89f31aa9bfd7df7a06"}
Oct 30 21:11:13.557450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.557319    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"
Oct 30 21:11:13.566592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.566545    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts" podStartSLOduration=1.652617501 podStartE2EDuration="4.566532652s" totalImagesPullingTime="2.913915151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:13.564834671 +0000 UTC m=+586.611393841" watchObservedRunningTime="2026-10-30 21:11:13.566532652 +0000 UTC m=+586.613091797"
Oct 30 21:11:13.586419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.586359    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg" podStartSLOduration=1.6288247930000002 podStartE2EDuration="4.586342582s" totalImagesPullingTime="2.957517789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:13.585436093 +0000 UTC m=+586.631995241" watchObservedRunningTime="2026-10-30 21:11:13.586342582 +0000 UTC m=+586.632901726"
Oct 30 21:11:13.611897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.611841    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w" podStartSLOduration=1.686835656 podStartE2EDuration="3.611828175s" totalImagesPullingTime="1.924992519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:13.610142794 +0000 UTC m=+586.656701952" watchObservedRunningTime="2026-10-30 21:11:13.611828175 +0000 UTC m=+586.658387320"
Oct 30 21:11:13.821963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.821896    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"]
Oct 30 21:11:13.825463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.825447    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.839305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.839287    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5ee9aa6f-openshift-default-dockercfg-2c4x8\""
Oct 30 21:11:13.852550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.852527    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"]
Oct 30 21:11:13.875108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.875081    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-envoy\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.875209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.875120    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-token\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.875209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.875154    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-socket\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.875295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.875269    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-podinfo\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.875441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.875423    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-certs\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.875487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.875469    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpc2\" (UniqueName: \"kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-kube-api-access-btpc2\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.875537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.875522    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-data\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.875611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.875596    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-credential-socket\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.875655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.875625    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istiod-ca-cert\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.979324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.976127    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-data\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.979324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.976181    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-credential-socket\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.979324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.976212    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istiod-ca-cert\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.979324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.976269    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-envoy\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.979324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.976318    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-token\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.979324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.976364    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-socket\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.979324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.976429    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-podinfo\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.979324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.976520    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-certs\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.979324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.976549    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpc2\" (UniqueName: \"kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-kube-api-access-btpc2\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.979324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.977423    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-data\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.979324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.977697    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-credential-socket\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.979324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.978371    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istiod-ca-cert\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.981067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.981041    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-socket\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.981310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.981281    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-certs\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.984853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.984827    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-podinfo\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:13.989049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:13.989026    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-envoy\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:14.000766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:14.000733    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpc2\" (UniqueName: \"kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-kube-api-access-btpc2\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:14.002333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:14.002305    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-token\") pod \"e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") " pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:14.138921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:14.138839    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:14.420512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:14.420489    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"]
Oct 30 21:11:14.421588 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:14.421559    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d48bcb0_6344_4b05_a92f_272ca78a9fdf.slice/crio-cabea3df06a88b5be405b21d62757bcc4e851d6854f408ecebb84afe9ef2aad4 WatchSource:0}: Error finding container cabea3df06a88b5be405b21d62757bcc4e851d6854f408ecebb84afe9ef2aad4: Status 404 returned error can't find the container with id cabea3df06a88b5be405b21d62757bcc4e851d6854f408ecebb84afe9ef2aad4
Oct 30 21:11:14.423832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:14.423803    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:14.423902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:14.423864    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:14.423902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:14.423892    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:14.564599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:14.564564    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" event={"ID":"7d48bcb0-6344-4b05-a92f-272ca78a9fdf","Type":"ContainerStarted","Data":"d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f"}
Oct 30 21:11:14.565072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:14.564657    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" event={"ID":"7d48bcb0-6344-4b05-a92f-272ca78a9fdf","Type":"ContainerStarted","Data":"cabea3df06a88b5be405b21d62757bcc4e851d6854f408ecebb84afe9ef2aad4"}
Oct 30 21:11:14.586730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:14.586681    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" podStartSLOduration=1.586669551 podStartE2EDuration="1.586669551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:14.586131609 +0000 UTC m=+587.632690771" watchObservedRunningTime="2026-10-30 21:11:14.586669551 +0000 UTC m=+587.633228690"
Oct 30 21:11:15.139454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:15.139423    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:15.144832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:15.144809    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:15.580688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:15.580650    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:15.582454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:15.582433    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:17.461536 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.461506    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"]
Oct 30 21:11:17.465115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.465100    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"
Oct 30 21:11:17.473090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.473071    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-5ee9aa6f-dockercfg-htk2q\""
Oct 30 21:11:17.474996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.474976    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-5ee9aa6f\""
Oct 30 21:11:17.477130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.477110    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"]
Oct 30 21:11:17.520056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.519980    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09b1342-195a-401d-a1bb-2827378fb501-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn\" (UID: \"e09b1342-195a-401d-a1bb-2827378fb501\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"
Oct 30 21:11:17.520241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.520130    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xzjn\" (UniqueName: \"kubernetes.io/projected/e09b1342-195a-401d-a1bb-2827378fb501-kube-api-access-9xzjn\") pod \"payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn\" (UID: \"e09b1342-195a-401d-a1bb-2827378fb501\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"
Oct 30 21:11:17.530068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.529053    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"]
Oct 30 21:11:17.536422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.533668    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"]
Oct 30 21:11:17.536422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.533744    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"
Oct 30 21:11:17.629427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.625989    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xzjn\" (UniqueName: \"kubernetes.io/projected/e09b1342-195a-401d-a1bb-2827378fb501-kube-api-access-9xzjn\") pod \"payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn\" (UID: \"e09b1342-195a-401d-a1bb-2827378fb501\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"
Oct 30 21:11:17.629427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.626067    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkdj7\" (UniqueName: \"kubernetes.io/projected/11880667-21f9-4cb0-973a-1a24773fd070-kube-api-access-fkdj7\") pod \"payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889\" (UID: \"11880667-21f9-4cb0-973a-1a24773fd070\") " pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"
Oct 30 21:11:17.629427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.626175    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09b1342-195a-401d-a1bb-2827378fb501-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn\" (UID: \"e09b1342-195a-401d-a1bb-2827378fb501\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"
Oct 30 21:11:17.629427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.626219    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11880667-21f9-4cb0-973a-1a24773fd070-plugins-config-volume\") pod \"payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889\" (UID: \"11880667-21f9-4cb0-973a-1a24773fd070\") " pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"
Oct 30 21:11:17.630150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.630105    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09b1342-195a-401d-a1bb-2827378fb501-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn\" (UID: \"e09b1342-195a-401d-a1bb-2827378fb501\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"
Oct 30 21:11:17.641116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.637593    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xzjn\" (UniqueName: \"kubernetes.io/projected/e09b1342-195a-401d-a1bb-2827378fb501-kube-api-access-9xzjn\") pod \"payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn\" (UID: \"e09b1342-195a-401d-a1bb-2827378fb501\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"
Oct 30 21:11:17.727160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.727129    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkdj7\" (UniqueName: \"kubernetes.io/projected/11880667-21f9-4cb0-973a-1a24773fd070-kube-api-access-fkdj7\") pod \"payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889\" (UID: \"11880667-21f9-4cb0-973a-1a24773fd070\") " pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"
Oct 30 21:11:17.727317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.727202    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11880667-21f9-4cb0-973a-1a24773fd070-plugins-config-volume\") pod \"payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889\" (UID: \"11880667-21f9-4cb0-973a-1a24773fd070\") " pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"
Oct 30 21:11:17.727928 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.727900    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11880667-21f9-4cb0-973a-1a24773fd070-plugins-config-volume\") pod \"payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889\" (UID: \"11880667-21f9-4cb0-973a-1a24773fd070\") " pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"
Oct 30 21:11:17.736221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.736194    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkdj7\" (UniqueName: \"kubernetes.io/projected/11880667-21f9-4cb0-973a-1a24773fd070-kube-api-access-fkdj7\") pod \"payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889\" (UID: \"11880667-21f9-4cb0-973a-1a24773fd070\") " pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"
Oct 30 21:11:17.778117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.778094    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"
Oct 30 21:11:17.847640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:17.847615    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"
Oct 30 21:11:18.077899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:18.077878    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"]
Oct 30 21:11:18.080349 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:18.080325    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09b1342_195a_401d_a1bb_2827378fb501.slice/crio-630f39b9c0fc27252856e1a3016d1c08e7adcd3d95184e69d53bd62aff77218a WatchSource:0}: Error finding container 630f39b9c0fc27252856e1a3016d1c08e7adcd3d95184e69d53bd62aff77218a: Status 404 returned error can't find the container with id 630f39b9c0fc27252856e1a3016d1c08e7adcd3d95184e69d53bd62aff77218a
Oct 30 21:11:18.163317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:18.163295    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"]
Oct 30 21:11:18.165016 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:18.164993    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11880667_21f9_4cb0_973a_1a24773fd070.slice/crio-80cb55a9f689d7eb63d8cd6c12e2889fb2de2a7e8923ceebfebcd47e4a24f390 WatchSource:0}: Error finding container 80cb55a9f689d7eb63d8cd6c12e2889fb2de2a7e8923ceebfebcd47e4a24f390: Status 404 returned error can't find the container with id 80cb55a9f689d7eb63d8cd6c12e2889fb2de2a7e8923ceebfebcd47e4a24f390
Oct 30 21:11:18.622274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:18.622237    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889" event={"ID":"11880667-21f9-4cb0-973a-1a24773fd070","Type":"ContainerStarted","Data":"8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e"}
Oct 30 21:11:18.622797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:18.622290    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889" event={"ID":"11880667-21f9-4cb0-973a-1a24773fd070","Type":"ContainerStarted","Data":"80cb55a9f689d7eb63d8cd6c12e2889fb2de2a7e8923ceebfebcd47e4a24f390"}
Oct 30 21:11:18.622797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:18.622331    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"
Oct 30 21:11:18.623878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:18.623849    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn" event={"ID":"e09b1342-195a-401d-a1bb-2827378fb501","Type":"ContainerStarted","Data":"16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee"}
Oct 30 21:11:18.623985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:18.623887    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn" event={"ID":"e09b1342-195a-401d-a1bb-2827378fb501","Type":"ContainerStarted","Data":"630f39b9c0fc27252856e1a3016d1c08e7adcd3d95184e69d53bd62aff77218a"}
Oct 30 21:11:18.623985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:18.623911    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"
Oct 30 21:11:18.640133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:18.640083    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889" podStartSLOduration=1.640068997 podStartE2EDuration="1.640068997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:18.638666041 +0000 UTC m=+591.685225185" watchObservedRunningTime="2026-10-30 21:11:18.640068997 +0000 UTC m=+591.686628144"
Oct 30 21:11:23.243070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.242998    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn" podStartSLOduration=6.242977436 podStartE2EDuration="6.242977436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:18.655483008 +0000 UTC m=+591.702042154" watchObservedRunningTime="2026-10-30 21:11:23.242977436 +0000 UTC m=+596.289536582"
Oct 30 21:11:23.245358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.245334    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"]
Oct 30 21:11:23.249810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.249787    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:23.257577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.257560    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:11:23.258081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.258054    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:11:23.258191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.258177    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:11:23.258256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.258249    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:11:23.258988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.258954    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"]
Oct 30 21:11:23.324799 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.324758    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf\" (UID: \"19cf8e47-1e22-4d58-886c-446094371e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:23.324799 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.324804    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mwrt\" (UniqueName: \"kubernetes.io/projected/19cf8e47-1e22-4d58-886c-446094371e2c-kube-api-access-6mwrt\") pod \"maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf\" (UID: \"19cf8e47-1e22-4d58-886c-446094371e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:23.325001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.324959    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf\" (UID: \"19cf8e47-1e22-4d58-886c-446094371e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:23.426274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.426228    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf\" (UID: \"19cf8e47-1e22-4d58-886c-446094371e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:23.426274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.426273    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mwrt\" (UniqueName: \"kubernetes.io/projected/19cf8e47-1e22-4d58-886c-446094371e2c-kube-api-access-6mwrt\") pod \"maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf\" (UID: \"19cf8e47-1e22-4d58-886c-446094371e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:23.426530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.426359    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf\" (UID: \"19cf8e47-1e22-4d58-886c-446094371e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:23.427065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.427041    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf\" (UID: \"19cf8e47-1e22-4d58-886c-446094371e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:23.429081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.429060    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf\" (UID: \"19cf8e47-1e22-4d58-886c-446094371e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:23.435091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.435063    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mwrt\" (UniqueName: \"kubernetes.io/projected/19cf8e47-1e22-4d58-886c-446094371e2c-kube-api-access-6mwrt\") pod \"maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf\" (UID: \"19cf8e47-1e22-4d58-886c-446094371e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:23.549625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.549533    4918 patch_prober.go:29] interesting pod/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:11:23.549752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.549612    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:11:23.567076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.567048    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:23.905931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:23.905846    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"]
Oct 30 21:11:23.920565 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:23.920525    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19cf8e47_1e22_4d58_886c_446094371e2c.slice/crio-f31d0efe2dd96225e7614e183e0063c92b8839e129befce232d48c9a2e678f9d WatchSource:0}: Error finding container f31d0efe2dd96225e7614e183e0063c92b8839e129befce232d48c9a2e678f9d: Status 404 returned error can't find the container with id f31d0efe2dd96225e7614e183e0063c92b8839e129befce232d48c9a2e678f9d
Oct 30 21:11:24.567585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:24.567559    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"
Oct 30 21:11:24.567967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:24.567623    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"
Oct 30 21:11:24.674761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:24.674715    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf" event={"ID":"19cf8e47-1e22-4d58-886c-446094371e2c","Type":"ContainerStarted","Data":"f31d0efe2dd96225e7614e183e0063c92b8839e129befce232d48c9a2e678f9d"}
Oct 30 21:11:25.583906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:25.583873    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"
Oct 30 21:11:27.114059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:27.114029    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:11:27.114746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:27.114725    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:11:27.695308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:27.695222    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf" event={"ID":"19cf8e47-1e22-4d58-886c-446094371e2c","Type":"ContainerStarted","Data":"4fc5f3eaf29f03374805b998e6106f89c53d013130726151851d681dd49b11d8"}
Oct 30 21:11:27.697032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:27.697007    4918 generic.go:356] "Generic (PLEG): container finished" podID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerID="7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9" exitCode=0
Oct 30 21:11:27.697181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:27.697124    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" event={"ID":"1db5e191-2a8e-40d2-819c-e9c00f33f07e","Type":"ContainerDied","Data":"7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9"}
Oct 30 21:11:27.697533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:27.697503    4918 scope.go:122] "RemoveContainer" containerID="7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9"
Oct 30 21:11:27.698903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:27.698875    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:27.698991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:27.698959    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:27.699063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:27.698993    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:27.738970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:27.738893    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf" podStartSLOduration=1.626943262 podStartE2EDuration="4.738873355s" totalImagesPullingTime="3.111930093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:27.715097931 +0000 UTC m=+600.761657089" watchObservedRunningTime="2026-10-30 21:11:27.738873355 +0000 UTC m=+600.785432500"
Oct 30 21:11:28.722244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:28.722212    4918 generic.go:356] "Generic (PLEG): container finished" podID="19cf8e47-1e22-4d58-886c-446094371e2c" containerID="4fc5f3eaf29f03374805b998e6106f89c53d013130726151851d681dd49b11d8" exitCode=0
Oct 30 21:11:28.722700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:28.722297    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf" event={"ID":"19cf8e47-1e22-4d58-886c-446094371e2c","Type":"ContainerDied","Data":"4fc5f3eaf29f03374805b998e6106f89c53d013130726151851d681dd49b11d8"}
Oct 30 21:11:28.724360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:28.724337    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" event={"ID":"1db5e191-2a8e-40d2-819c-e9c00f33f07e","Type":"ContainerStarted","Data":"ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0"}
Oct 30 21:11:29.055949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.055857    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"]
Oct 30 21:11:29.062805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.062778    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:29.067750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.067726    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"]
Oct 30 21:11:29.125906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.125861    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr25g\" (UniqueName: \"kubernetes.io/projected/d1c994cd-4341-46e1-ad91-303bbfbda1aa-kube-api-access-sr25g\") pod \"maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh\" (UID: \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:29.126083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.125920    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh\" (UID: \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:29.126083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.126046    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh\" (UID: \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:29.159030 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:11:29.158988    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:11:29.226969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.226931    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr25g\" (UniqueName: \"kubernetes.io/projected/d1c994cd-4341-46e1-ad91-303bbfbda1aa-kube-api-access-sr25g\") pod \"maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh\" (UID: \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:29.227141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.226982    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh\" (UID: \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:29.227141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.227089    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh\" (UID: \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:29.227683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.227648    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh\" (UID: \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:29.230076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.230049    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh\" (UID: \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:29.236090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.236063    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr25g\" (UniqueName: \"kubernetes.io/projected/d1c994cd-4341-46e1-ad91-303bbfbda1aa-kube-api-access-sr25g\") pod \"maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh\" (UID: \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:29.380821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.380746    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:29.527006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.526972    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:29.527233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.527007    4918 patch_prober.go:29] interesting pod/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Oct 30 21:11:29.527233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.527052    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Oct 30 21:11:29.628656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.628620    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"]
Oct 30 21:11:29.632237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.632182    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"
Oct 30 21:11:29.632417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.632381    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:29.632464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.632444    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"
Oct 30 21:11:29.640230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.640013    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"]
Oct 30 21:11:29.684724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.684597    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"]
Oct 30 21:11:29.713988 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:29.713952    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c994cd_4341_46e1_ad91_303bbfbda1aa.slice/crio-d39aefbd195c3bdc2b24aa54e98e2ac4c232a74b67161666fde7ca19ac504a43 WatchSource:0}: Error finding container d39aefbd195c3bdc2b24aa54e98e2ac4c232a74b67161666fde7ca19ac504a43: Status 404 returned error can't find the container with id d39aefbd195c3bdc2b24aa54e98e2ac4c232a74b67161666fde7ca19ac504a43
Oct 30 21:11:29.731125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.731090    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8xnx\" (UniqueName: \"kubernetes.io/projected/aacda18d-c8bb-4c11-ad4f-c74f12b62798-kube-api-access-k8xnx\") pod \"maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs\" (UID: \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:29.731586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.731243    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs\" (UID: \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:29.731586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.731454    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs\" (UID: \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:29.733332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.733308    4918 generic.go:356] "Generic (PLEG): container finished" podID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerID="0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4" exitCode=0
Oct 30 21:11:29.733465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.733417    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" event={"ID":"f24d688c-575f-4c80-8745-8cc0d72ad30c","Type":"ContainerDied","Data":"0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4"}
Oct 30 21:11:29.733923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.733830    4918 scope.go:122] "RemoveContainer" containerID="0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4"
Oct 30 21:11:29.735467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.735442    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh" event={"ID":"d1c994cd-4341-46e1-ad91-303bbfbda1aa","Type":"ContainerStarted","Data":"d39aefbd195c3bdc2b24aa54e98e2ac4c232a74b67161666fde7ca19ac504a43"}
Oct 30 21:11:29.735600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.735445    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:29.735600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.735549    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:29.735714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.735601    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:29.833054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.833023    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs\" (UID: \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:29.833170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.833109    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs\" (UID: \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:29.833406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.833370    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8xnx\" (UniqueName: \"kubernetes.io/projected/aacda18d-c8bb-4c11-ad4f-c74f12b62798-kube-api-access-k8xnx\") pod \"maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs\" (UID: \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:29.834304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.834281    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs\" (UID: \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:29.835983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.835959    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs\" (UID: \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:29.843056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.843030    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8xnx\" (UniqueName: \"kubernetes.io/projected/aacda18d-c8bb-4c11-ad4f-c74f12b62798-kube-api-access-k8xnx\") pod \"maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs\" (UID: \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:29.949422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.947463    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:29.957421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:29.957400    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:30.137846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.137802    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-openshift-ca\") pod \"19cf8e47-1e22-4d58-886c-446094371e2c\" (UID: \"19cf8e47-1e22-4d58-886c-446094371e2c\") "
Oct 30 21:11:30.138131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.138118    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19cf8e47-1e22-4d58-886c-446094371e2c-kube-api-access-6mwrt\" (UniqueName: \"kubernetes.io/projected/19cf8e47-1e22-4d58-886c-446094371e2c-kube-api-access-6mwrt\") pod \"19cf8e47-1e22-4d58-886c-446094371e2c\" (UID: \"19cf8e47-1e22-4d58-886c-446094371e2c\") "
Oct 30 21:11:30.138189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.138145    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-xks-ca\") pod \"19cf8e47-1e22-4d58-886c-446094371e2c\" (UID: \"19cf8e47-1e22-4d58-886c-446094371e2c\") "
Oct 30 21:11:30.138189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.138161    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-openshift-ca" pod "19cf8e47-1e22-4d58-886c-446094371e2c" (UID: "19cf8e47-1e22-4d58-886c-446094371e2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:30.138451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.138437    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:30.140896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.140862    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19cf8e47-1e22-4d58-886c-446094371e2c-kube-api-access-6mwrt" pod "19cf8e47-1e22-4d58-886c-446094371e2c" (UID: "19cf8e47-1e22-4d58-886c-446094371e2c"). InnerVolumeSpecName "kube-api-access-6mwrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:30.141136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.141110    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-xks-ca" pod "19cf8e47-1e22-4d58-886c-446094371e2c" (UID: "19cf8e47-1e22-4d58-886c-446094371e2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:30.238988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.238960    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mwrt\" (UniqueName: \"kubernetes.io/projected/19cf8e47-1e22-4d58-886c-446094371e2c-kube-api-access-6mwrt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:30.238988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.238985    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19cf8e47-1e22-4d58-886c-446094371e2c-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:30.266936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.266901    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"]
Oct 30 21:11:30.542199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.541525    4918 patch_prober.go:29] interesting pod/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Oct 30 21:11:30.542199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.541597    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Oct 30 21:11:30.745797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.745762    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs" event={"ID":"aacda18d-c8bb-4c11-ad4f-c74f12b62798","Type":"ContainerStarted","Data":"b94e4c8f9ca6798d87d3d5bb8ca443e73b5c1888aac17ba5e1f68e3a8c207c06"}
Oct 30 21:11:30.747270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.747250    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"
Oct 30 21:11:30.747357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.747263    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf" event={"ID":"19cf8e47-1e22-4d58-886c-446094371e2c","Type":"ContainerDied","Data":"f31d0efe2dd96225e7614e183e0063c92b8839e129befce232d48c9a2e678f9d"}
Oct 30 21:11:30.747357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.747305    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f31d0efe2dd96225e7614e183e0063c92b8839e129befce232d48c9a2e678f9d"
Oct 30 21:11:30.751588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.751558    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" event={"ID":"f24d688c-575f-4c80-8745-8cc0d72ad30c","Type":"ContainerStarted","Data":"6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df"}
Oct 30 21:11:30.753180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.753158    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh" event={"ID":"d1c994cd-4341-46e1-ad91-303bbfbda1aa","Type":"ContainerStarted","Data":"409570eb85ef697b7a928b8c2f0264f17baec804ce69d2cde6fa3722ba0ddcf0"}
Oct 30 21:11:30.816179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:30.816120    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh" podStartSLOduration=1.8161018759999998 podStartE2EDuration="1.816101876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:30.804986062 +0000 UTC m=+603.851545233" watchObservedRunningTime="2026-10-30 21:11:30.816101876 +0000 UTC m=+603.862661021"
Oct 30 21:11:31.260213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.254680    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"]
Oct 30 21:11:31.260213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.255090    4918 cpu_manager.go:414] "containerMap: removing container" podUID="19cf8e47-1e22-4d58-886c-446094371e2c" containerName="revoke-keys"
Oct 30 21:11:31.260213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.255100    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19cf8e47-1e22-4d58-886c-446094371e2c" containerName="revoke-keys"
Oct 30 21:11:31.260213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.255182    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="19cf8e47-1e22-4d58-886c-446094371e2c" containerName="revoke-keys"
Oct 30 21:11:31.260792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.260770    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"]
Oct 30 21:11:31.261062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.260999    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:31.327416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.327052    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:31.327416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.327184    4918 patch_prober.go:29] interesting pod/e2e-ait-111fd691-openshift-default-598486d85f-njfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:11:31.327416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.327230    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 21:11:31.361419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.360518    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13506529-7fda-4f22-8b20-0df078337c21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-111fd691-lbtsc\" (UID: \"13506529-7fda-4f22-8b20-0df078337c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:31.361419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.360575    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13506529-7fda-4f22-8b20-0df078337c21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-111fd691-lbtsc\" (UID: \"13506529-7fda-4f22-8b20-0df078337c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:31.361419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.360649    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxg97\" (UniqueName: \"kubernetes.io/projected/13506529-7fda-4f22-8b20-0df078337c21-kube-api-access-vxg97\") pod \"maas-api-revoke-keys-e2e-ait-111fd691-lbtsc\" (UID: \"13506529-7fda-4f22-8b20-0df078337c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:31.462428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.462353    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxg97\" (UniqueName: \"kubernetes.io/projected/13506529-7fda-4f22-8b20-0df078337c21-kube-api-access-vxg97\") pod \"maas-api-revoke-keys-e2e-ait-111fd691-lbtsc\" (UID: \"13506529-7fda-4f22-8b20-0df078337c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:31.462877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.462807    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13506529-7fda-4f22-8b20-0df078337c21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-111fd691-lbtsc\" (UID: \"13506529-7fda-4f22-8b20-0df078337c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:31.463037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.462850    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13506529-7fda-4f22-8b20-0df078337c21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-111fd691-lbtsc\" (UID: \"13506529-7fda-4f22-8b20-0df078337c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:31.464506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.464468    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13506529-7fda-4f22-8b20-0df078337c21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-111fd691-lbtsc\" (UID: \"13506529-7fda-4f22-8b20-0df078337c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:31.468157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.468041    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13506529-7fda-4f22-8b20-0df078337c21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-111fd691-lbtsc\" (UID: \"13506529-7fda-4f22-8b20-0df078337c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:31.495416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.477245    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxg97\" (UniqueName: \"kubernetes.io/projected/13506529-7fda-4f22-8b20-0df078337c21-kube-api-access-vxg97\") pod \"maas-api-revoke-keys-e2e-ait-111fd691-lbtsc\" (UID: \"13506529-7fda-4f22-8b20-0df078337c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:31.528473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.527855    4918 patch_prober.go:29] interesting pod/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Oct 30 21:11:31.528473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.527917    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Oct 30 21:11:31.578632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.578597    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:31.769716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.769685    4918 generic.go:356] "Generic (PLEG): container finished" podID="d1c994cd-4341-46e1-ad91-303bbfbda1aa" containerID="409570eb85ef697b7a928b8c2f0264f17baec804ce69d2cde6fa3722ba0ddcf0" exitCode=0
Oct 30 21:11:31.770198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.769770    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh" event={"ID":"d1c994cd-4341-46e1-ad91-303bbfbda1aa","Type":"ContainerDied","Data":"409570eb85ef697b7a928b8c2f0264f17baec804ce69d2cde6fa3722ba0ddcf0"}
Oct 30 21:11:31.771375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.771350    4918 generic.go:356] "Generic (PLEG): container finished" podID="aacda18d-c8bb-4c11-ad4f-c74f12b62798" containerID="11d4fd0f5d627048687c4d621ffe082c9fb129e5fb2c40495f92d773b080ee33" exitCode=0
Oct 30 21:11:31.771537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:31.771427    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs" event={"ID":"aacda18d-c8bb-4c11-ad4f-c74f12b62798","Type":"ContainerDied","Data":"11d4fd0f5d627048687c4d621ffe082c9fb129e5fb2c40495f92d773b080ee33"}
Oct 30 21:11:32.073975 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:32.073936    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13506529_7fda_4f22_8b20_0df078337c21.slice/crio-be3f223561a1740769aa91557a81b9c7318f49a4f8d542b8ce36081a94d49bb6 WatchSource:0}: Error finding container be3f223561a1740769aa91557a81b9c7318f49a4f8d542b8ce36081a94d49bb6: Status 404 returned error can't find the container with id be3f223561a1740769aa91557a81b9c7318f49a4f8d542b8ce36081a94d49bb6
Oct 30 21:11:32.074354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:32.074330    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"]
Oct 30 21:11:32.309415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:32.309267    4918 patch_prober.go:29] interesting pod/e2e-ait-111fd691-openshift-default-598486d85f-njfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:11:32.309415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:32.309331    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 21:11:32.539674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:32.538778    4918 patch_prober.go:29] interesting pod/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Oct 30 21:11:32.539674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:32.538831    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Oct 30 21:11:32.778680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:32.778643    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc" event={"ID":"13506529-7fda-4f22-8b20-0df078337c21","Type":"ContainerStarted","Data":"0f22c44141db262777a5db6f20c9da19abb2822938bf23b06bbf4dd1a764d352"}
Oct 30 21:11:32.779113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:32.778706    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc" event={"ID":"13506529-7fda-4f22-8b20-0df078337c21","Type":"ContainerStarted","Data":"be3f223561a1740769aa91557a81b9c7318f49a4f8d542b8ce36081a94d49bb6"}
Oct 30 21:11:32.838417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:32.822842    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc" podStartSLOduration=1.822824875 podStartE2EDuration="1.822824875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:32.820048694 +0000 UTC m=+605.866607840" watchObservedRunningTime="2026-10-30 21:11:32.822824875 +0000 UTC m=+605.869384022"
Oct 30 21:11:33.336476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.336441    4918 patch_prober.go:29] interesting pod/e2e-ait-111fd691-openshift-default-598486d85f-njfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:11:33.345450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.336695    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 21:11:33.532417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.531499    4918 patch_prober.go:29] interesting pod/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Oct 30 21:11:33.532417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.531560    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Oct 30 21:11:33.597427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.592160    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:33.694071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.693999    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-xks-ca\") pod \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\" (UID: \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\") "
Oct 30 21:11:33.694722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.694661    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c994cd-4341-46e1-ad91-303bbfbda1aa-kube-api-access-sr25g\" (UniqueName: \"kubernetes.io/projected/d1c994cd-4341-46e1-ad91-303bbfbda1aa-kube-api-access-sr25g\") pod \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\" (UID: \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\") "
Oct 30 21:11:33.694929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.694889    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-openshift-ca\") pod \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\" (UID: \"d1c994cd-4341-46e1-ad91-303bbfbda1aa\") "
Oct 30 21:11:33.739578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.724946    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-openshift-ca" pod "d1c994cd-4341-46e1-ad91-303bbfbda1aa" (UID: "d1c994cd-4341-46e1-ad91-303bbfbda1aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:33.756686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.754985    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-xks-ca" pod "d1c994cd-4341-46e1-ad91-303bbfbda1aa" (UID: "d1c994cd-4341-46e1-ad91-303bbfbda1aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:33.777412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.774549    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c994cd-4341-46e1-ad91-303bbfbda1aa-kube-api-access-sr25g" pod "d1c994cd-4341-46e1-ad91-303bbfbda1aa" (UID: "d1c994cd-4341-46e1-ad91-303bbfbda1aa"). InnerVolumeSpecName "kube-api-access-sr25g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:33.785198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.785122    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:33.844415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.841677    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr25g\" (UniqueName: \"kubernetes.io/projected/d1c994cd-4341-46e1-ad91-303bbfbda1aa-kube-api-access-sr25g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:33.844415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.841712    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:33.844415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.841728    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c994cd-4341-46e1-ad91-303bbfbda1aa-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:33.854764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.854302    4918 generic.go:356] "Generic (PLEG): container finished" podID="13506529-7fda-4f22-8b20-0df078337c21" containerID="0f22c44141db262777a5db6f20c9da19abb2822938bf23b06bbf4dd1a764d352" exitCode=0
Oct 30 21:11:33.854764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.854479    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc" event={"ID":"13506529-7fda-4f22-8b20-0df078337c21","Type":"ContainerDied","Data":"0f22c44141db262777a5db6f20c9da19abb2822938bf23b06bbf4dd1a764d352"}
Oct 30 21:11:33.859410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.856117    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh" event={"ID":"d1c994cd-4341-46e1-ad91-303bbfbda1aa","Type":"ContainerDied","Data":"d39aefbd195c3bdc2b24aa54e98e2ac4c232a74b67161666fde7ca19ac504a43"}
Oct 30 21:11:33.859410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.856151    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d39aefbd195c3bdc2b24aa54e98e2ac4c232a74b67161666fde7ca19ac504a43"
Oct 30 21:11:33.859410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.856199    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"
Oct 30 21:11:33.865727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.865702    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs" event={"ID":"aacda18d-c8bb-4c11-ad4f-c74f12b62798","Type":"ContainerDied","Data":"b94e4c8f9ca6798d87d3d5bb8ca443e73b5c1888aac17ba5e1f68e3a8c207c06"}
Oct 30 21:11:33.865850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.865748    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b94e4c8f9ca6798d87d3d5bb8ca443e73b5c1888aac17ba5e1f68e3a8c207c06"
Oct 30 21:11:33.865850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.865824    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"
Oct 30 21:11:33.948898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.948862    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-xks-ca\") pod \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\" (UID: \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\") "
Oct 30 21:11:33.949218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.949195    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacda18d-c8bb-4c11-ad4f-c74f12b62798-kube-api-access-k8xnx\" (UniqueName: \"kubernetes.io/projected/aacda18d-c8bb-4c11-ad4f-c74f12b62798-kube-api-access-k8xnx\") pod \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\" (UID: \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\") "
Oct 30 21:11:33.949473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.949452    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-openshift-ca\") pod \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\" (UID: \"aacda18d-c8bb-4c11-ad4f-c74f12b62798\") "
Oct 30 21:11:33.962721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.951108    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-openshift-ca" pod "aacda18d-c8bb-4c11-ad4f-c74f12b62798" (UID: "aacda18d-c8bb-4c11-ad4f-c74f12b62798"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:33.963026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.962994    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:33.963828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.963781    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacda18d-c8bb-4c11-ad4f-c74f12b62798-kube-api-access-k8xnx" pod "aacda18d-c8bb-4c11-ad4f-c74f12b62798" (UID: "aacda18d-c8bb-4c11-ad4f-c74f12b62798"). InnerVolumeSpecName "kube-api-access-k8xnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:33.965979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:33.965955    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-xks-ca" pod "aacda18d-c8bb-4c11-ad4f-c74f12b62798" (UID: "aacda18d-c8bb-4c11-ad4f-c74f12b62798"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:34.081466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:34.068572    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8xnx\" (UniqueName: \"kubernetes.io/projected/aacda18d-c8bb-4c11-ad4f-c74f12b62798-kube-api-access-k8xnx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:34.081466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:34.068620    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacda18d-c8bb-4c11-ad4f-c74f12b62798-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:34.310837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:34.310800    4918 patch_prober.go:29] interesting pod/e2e-ait-111fd691-openshift-default-598486d85f-njfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:11:34.311119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:34.311091    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 21:11:34.538699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:34.537666    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:34.538699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:34.538109    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:34.539088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:34.539072    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:35.150860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.150790    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:35.295418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.285499    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13506529-7fda-4f22-8b20-0df078337c21-kube-api-access-vxg97\" (UniqueName: \"kubernetes.io/projected/13506529-7fda-4f22-8b20-0df078337c21-kube-api-access-vxg97\") pod \"13506529-7fda-4f22-8b20-0df078337c21\" (UID: \"13506529-7fda-4f22-8b20-0df078337c21\") "
Oct 30 21:11:35.295418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.285597    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13506529-7fda-4f22-8b20-0df078337c21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13506529-7fda-4f22-8b20-0df078337c21-maas-api-openshift-ca\") pod \"13506529-7fda-4f22-8b20-0df078337c21\" (UID: \"13506529-7fda-4f22-8b20-0df078337c21\") "
Oct 30 21:11:35.295418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.285632    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13506529-7fda-4f22-8b20-0df078337c21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13506529-7fda-4f22-8b20-0df078337c21-maas-api-xks-ca\") pod \"13506529-7fda-4f22-8b20-0df078337c21\" (UID: \"13506529-7fda-4f22-8b20-0df078337c21\") "
Oct 30 21:11:35.300981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.300942    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13506529-7fda-4f22-8b20-0df078337c21-maas-api-openshift-ca" pod "13506529-7fda-4f22-8b20-0df078337c21" (UID: "13506529-7fda-4f22-8b20-0df078337c21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:35.328754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.328726    4918 patch_prober.go:29] interesting pod/e2e-ait-111fd691-openshift-default-598486d85f-njfwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:11:35.328939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.328912    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 30 21:11:35.329945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.329910    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13506529-7fda-4f22-8b20-0df078337c21-kube-api-access-vxg97" pod "13506529-7fda-4f22-8b20-0df078337c21" (UID: "13506529-7fda-4f22-8b20-0df078337c21"). InnerVolumeSpecName "kube-api-access-vxg97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:35.339542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.339514    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13506529-7fda-4f22-8b20-0df078337c21-maas-api-xks-ca" pod "13506529-7fda-4f22-8b20-0df078337c21" (UID: "13506529-7fda-4f22-8b20-0df078337c21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:35.400415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.395041    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxg97\" (UniqueName: \"kubernetes.io/projected/13506529-7fda-4f22-8b20-0df078337c21-kube-api-access-vxg97\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:35.400415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.395073    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13506529-7fda-4f22-8b20-0df078337c21-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:35.400415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.395094    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13506529-7fda-4f22-8b20-0df078337c21-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:35.877360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.877328    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"
Oct 30 21:11:35.877546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.877356    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc" event={"ID":"13506529-7fda-4f22-8b20-0df078337c21","Type":"ContainerDied","Data":"be3f223561a1740769aa91557a81b9c7318f49a4f8d542b8ce36081a94d49bb6"}
Oct 30 21:11:35.877546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:35.877429    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be3f223561a1740769aa91557a81b9c7318f49a4f8d542b8ce36081a94d49bb6"
Oct 30 21:11:36.175023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.174922    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"]
Oct 30 21:11:36.175533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.175362    4918 cpu_manager.go:414] "containerMap: removing container" podUID="aacda18d-c8bb-4c11-ad4f-c74f12b62798" containerName="revoke-keys"
Oct 30 21:11:36.175533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.175374    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacda18d-c8bb-4c11-ad4f-c74f12b62798" containerName="revoke-keys"
Oct 30 21:11:36.175533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.175422    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d1c994cd-4341-46e1-ad91-303bbfbda1aa" containerName="revoke-keys"
Oct 30 21:11:36.175533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.175430    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c994cd-4341-46e1-ad91-303bbfbda1aa" containerName="revoke-keys"
Oct 30 21:11:36.175533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.175449    4918 cpu_manager.go:414] "containerMap: removing container" podUID="13506529-7fda-4f22-8b20-0df078337c21" containerName="revoke-keys"
Oct 30 21:11:36.175533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.175454    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13506529-7fda-4f22-8b20-0df078337c21" containerName="revoke-keys"
Oct 30 21:11:36.175533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.175519    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacda18d-c8bb-4c11-ad4f-c74f12b62798" containerName="revoke-keys"
Oct 30 21:11:36.175533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.175528    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c994cd-4341-46e1-ad91-303bbfbda1aa" containerName="revoke-keys"
Oct 30 21:11:36.175533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.175535    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="13506529-7fda-4f22-8b20-0df078337c21" containerName="revoke-keys"
Oct 30 21:11:36.179344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.179324    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:36.184441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.184313    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"]
Oct 30 21:11:36.185031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.185013    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:11:36.195807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.185814    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:11:36.196609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.196586    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:11:36.196917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.196898    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:11:36.314402 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.314350    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:36.314690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.314669    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:36.315452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.315434    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:36.330620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.330592    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t\" (UID: \"9f467c15-a3a0-44cd-81cf-52d80a253a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:36.330717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.330663    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t\" (UID: \"9f467c15-a3a0-44cd-81cf-52d80a253a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:36.330787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.330768    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88xpx\" (UniqueName: \"kubernetes.io/projected/9f467c15-a3a0-44cd-81cf-52d80a253a58-kube-api-access-88xpx\") pod \"maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t\" (UID: \"9f467c15-a3a0-44cd-81cf-52d80a253a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:36.431963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.431875    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t\" (UID: \"9f467c15-a3a0-44cd-81cf-52d80a253a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:36.432122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.431986    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t\" (UID: \"9f467c15-a3a0-44cd-81cf-52d80a253a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:36.432122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.432079    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88xpx\" (UniqueName: \"kubernetes.io/projected/9f467c15-a3a0-44cd-81cf-52d80a253a58-kube-api-access-88xpx\") pod \"maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t\" (UID: \"9f467c15-a3a0-44cd-81cf-52d80a253a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:36.432624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.432600    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t\" (UID: \"9f467c15-a3a0-44cd-81cf-52d80a253a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:36.434675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.434656    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t\" (UID: \"9f467c15-a3a0-44cd-81cf-52d80a253a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:36.440606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.440581    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88xpx\" (UniqueName: \"kubernetes.io/projected/9f467c15-a3a0-44cd-81cf-52d80a253a58-kube-api-access-88xpx\") pod \"maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t\" (UID: \"9f467c15-a3a0-44cd-81cf-52d80a253a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:36.512230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.512188    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"]
Oct 30 21:11:36.514927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.514885    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e239cab2-f64sf"]
Oct 30 21:11:36.533436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.533381    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:36.877817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:36.877793    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"]
Oct 30 21:11:36.879996 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:36.879963    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f467c15_a3a0_44cd_81cf_52d80a253a58.slice/crio-78c30e6d0eb56127c33260e2885fc2180facce21728cea84f208c366f8e47deb WatchSource:0}: Error finding container 78c30e6d0eb56127c33260e2885fc2180facce21728cea84f208c366f8e47deb: Status 404 returned error can't find the container with id 78c30e6d0eb56127c33260e2885fc2180facce21728cea84f208c366f8e47deb
Oct 30 21:11:37.076243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:37.076214    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19cf8e47-1e22-4d58-886c-446094371e2c" path="/var/lib/kubelet/pods/19cf8e47-1e22-4d58-886c-446094371e2c/volumes"
Oct 30 21:11:37.890862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:37.890820    4918 generic.go:356] "Generic (PLEG): container finished" podID="9f467c15-a3a0-44cd-81cf-52d80a253a58" containerID="eda9a3b324c9ec7a5f26070f9cf98c2c629d5ee61bf7001dc235495297eba85b" exitCode=0
Oct 30 21:11:37.891255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:37.890908    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t" event={"ID":"9f467c15-a3a0-44cd-81cf-52d80a253a58","Type":"ContainerDied","Data":"eda9a3b324c9ec7a5f26070f9cf98c2c629d5ee61bf7001dc235495297eba85b"}
Oct 30 21:11:37.891255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:37.890988    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t" event={"ID":"9f467c15-a3a0-44cd-81cf-52d80a253a58","Type":"ContainerStarted","Data":"78c30e6d0eb56127c33260e2885fc2180facce21728cea84f208c366f8e47deb"}
Oct 30 21:11:39.054462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.054439    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:39.160957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.160876    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-xks-ca\") pod \"9f467c15-a3a0-44cd-81cf-52d80a253a58\" (UID: \"9f467c15-a3a0-44cd-81cf-52d80a253a58\") "
Oct 30 21:11:39.160957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.160929    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f467c15-a3a0-44cd-81cf-52d80a253a58-kube-api-access-88xpx\" (UniqueName: \"kubernetes.io/projected/9f467c15-a3a0-44cd-81cf-52d80a253a58-kube-api-access-88xpx\") pod \"9f467c15-a3a0-44cd-81cf-52d80a253a58\" (UID: \"9f467c15-a3a0-44cd-81cf-52d80a253a58\") "
Oct 30 21:11:39.161145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.160994    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-openshift-ca\") pod \"9f467c15-a3a0-44cd-81cf-52d80a253a58\" (UID: \"9f467c15-a3a0-44cd-81cf-52d80a253a58\") "
Oct 30 21:11:39.161504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.161471    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-openshift-ca" pod "9f467c15-a3a0-44cd-81cf-52d80a253a58" (UID: "9f467c15-a3a0-44cd-81cf-52d80a253a58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:39.163231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.163208    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f467c15-a3a0-44cd-81cf-52d80a253a58-kube-api-access-88xpx" pod "9f467c15-a3a0-44cd-81cf-52d80a253a58" (UID: "9f467c15-a3a0-44cd-81cf-52d80a253a58"). InnerVolumeSpecName "kube-api-access-88xpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:39.163231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.163211    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-xks-ca" pod "9f467c15-a3a0-44cd-81cf-52d80a253a58" (UID: "9f467c15-a3a0-44cd-81cf-52d80a253a58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:11:39.222864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.222828    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"]
Oct 30 21:11:39.237312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.237284    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"]
Oct 30 21:11:39.262728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.262699    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:39.262728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.262728    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88xpx\" (UniqueName: \"kubernetes.io/projected/9f467c15-a3a0-44cd-81cf-52d80a253a58-kube-api-access-88xpx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:39.262939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.262742    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f467c15-a3a0-44cd-81cf-52d80a253a58-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:39.739262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.739223    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"]
Oct 30 21:11:39.915094 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.915061    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"
Oct 30 21:11:39.915249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.915101    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t" event={"ID":"9f467c15-a3a0-44cd-81cf-52d80a253a58","Type":"ContainerDied","Data":"78c30e6d0eb56127c33260e2885fc2180facce21728cea84f208c366f8e47deb"}
Oct 30 21:11:39.915249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:39.915162    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78c30e6d0eb56127c33260e2885fc2180facce21728cea84f208c366f8e47deb"
Oct 30 21:11:40.922119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:40.922087    4918 generic.go:356] "Generic (PLEG): container finished" podID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerID="d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f" exitCode=0
Oct 30 21:11:40.922576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:40.922162    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" event={"ID":"7d48bcb0-6344-4b05-a92f-272ca78a9fdf","Type":"ContainerDied","Data":"d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f"}
Oct 30 21:11:40.922703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:40.922686    4918 scope.go:122] "RemoveContainer" containerID="d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f"
Oct 30 21:11:40.924312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:40.924277    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:40.924464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:40.924363    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:40.924464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:40.924432    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:41.023423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:41.023377    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"]
Oct 30 21:11:41.036247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:41.036217    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e2cef583-bq6zh"]
Oct 30 21:11:41.078043 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:41.078010    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c994cd-4341-46e1-ad91-303bbfbda1aa" path="/var/lib/kubelet/pods/d1c994cd-4341-46e1-ad91-303bbfbda1aa/volumes"
Oct 30 21:11:41.196969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:41.196878    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"]
Oct 30 21:11:41.197163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:41.197128    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy" containerID="cri-o://ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0" gracePeriod=30
Oct 30 21:11:41.538523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:41.538485    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"]
Oct 30 21:11:41.539597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:41.539568    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6eedd82b-mb7qs"]
Oct 30 21:11:41.929600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:41.929512    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" event={"ID":"7d48bcb0-6344-4b05-a92f-272ca78a9fdf","Type":"ContainerStarted","Data":"8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30"}
Oct 30 21:11:42.139354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:42.139313    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:42.140773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:42.140743    4918 patch_prober.go:29] interesting pod/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 21:11:42.140923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:42.140794    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 21:11:42.963046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:42.963015    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"]
Oct 30 21:11:42.966680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:42.966654    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-111fd691-lbtsc"]
Oct 30 21:11:43.080693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:43.080664    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13506529-7fda-4f22-8b20-0df078337c21" path="/var/lib/kubelet/pods/13506529-7fda-4f22-8b20-0df078337c21/volumes"
Oct 30 21:11:43.081119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:43.081099    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacda18d-c8bb-4c11-ad4f-c74f12b62798" path="/var/lib/kubelet/pods/aacda18d-c8bb-4c11-ad4f-c74f12b62798/volumes"
Oct 30 21:11:43.139910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:43.139876    4918 patch_prober.go:29] interesting pod/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 21:11:43.140078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:43.139935    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 21:11:43.670426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:43.669655    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"]
Oct 30 21:11:43.670426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:43.669908    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerName="istio-proxy" containerID="cri-o://6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df" gracePeriod=30
Oct 30 21:11:44.140525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.140471    4918 patch_prober.go:29] interesting pod/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 21:11:44.140887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.140555    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 21:11:44.223230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.223178    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg" podUID="f10f3396-84b6-4fc8-ae81-94802cd42e09" containerName="payload-processing" containerID="cri-o://e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48" gracePeriod=25
Oct 30 21:11:44.237968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.237618    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts" podUID="2ea25b16-f21e-444d-a4fd-0faff56edf63" containerName="payload-pre-processing" containerID="cri-o://c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3" gracePeriod=25
Oct 30 21:11:44.565241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.565221    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"
Oct 30 21:11:44.569093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.569075    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"
Oct 30 21:11:44.651448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.651053    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea25b16-f21e-444d-a4fd-0faff56edf63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea25b16-f21e-444d-a4fd-0faff56edf63-plugins-config-volume\") pod \"2ea25b16-f21e-444d-a4fd-0faff56edf63\" (UID: \"2ea25b16-f21e-444d-a4fd-0faff56edf63\") "
Oct 30 21:11:44.651448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.651138    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea25b16-f21e-444d-a4fd-0faff56edf63-kube-api-access-58256\" (UniqueName: \"kubernetes.io/projected/2ea25b16-f21e-444d-a4fd-0faff56edf63-kube-api-access-58256\") pod \"2ea25b16-f21e-444d-a4fd-0faff56edf63\" (UID: \"2ea25b16-f21e-444d-a4fd-0faff56edf63\") "
Oct 30 21:11:44.651448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.651234    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10f3396-84b6-4fc8-ae81-94802cd42e09-kube-api-access-zt624\" (UniqueName: \"kubernetes.io/projected/f10f3396-84b6-4fc8-ae81-94802cd42e09-kube-api-access-zt624\") pod \"f10f3396-84b6-4fc8-ae81-94802cd42e09\" (UID: \"f10f3396-84b6-4fc8-ae81-94802cd42e09\") "
Oct 30 21:11:44.651448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.651272    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10f3396-84b6-4fc8-ae81-94802cd42e09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10f3396-84b6-4fc8-ae81-94802cd42e09-plugins-config-volume\") pod \"f10f3396-84b6-4fc8-ae81-94802cd42e09\" (UID: \"f10f3396-84b6-4fc8-ae81-94802cd42e09\") "
Oct 30 21:11:44.656365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.652826    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea25b16-f21e-444d-a4fd-0faff56edf63-plugins-config-volume" pod "2ea25b16-f21e-444d-a4fd-0faff56edf63" (UID: "2ea25b16-f21e-444d-a4fd-0faff56edf63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:44.658410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.658349    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea25b16-f21e-444d-a4fd-0faff56edf63-kube-api-access-58256" pod "2ea25b16-f21e-444d-a4fd-0faff56edf63" (UID: "2ea25b16-f21e-444d-a4fd-0faff56edf63"). InnerVolumeSpecName "kube-api-access-58256". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:44.659409 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.659029    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10f3396-84b6-4fc8-ae81-94802cd42e09-kube-api-access-zt624" pod "f10f3396-84b6-4fc8-ae81-94802cd42e09" (UID: "f10f3396-84b6-4fc8-ae81-94802cd42e09"). InnerVolumeSpecName "kube-api-access-zt624". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:44.660693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.660669    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10f3396-84b6-4fc8-ae81-94802cd42e09-plugins-config-volume" pod "f10f3396-84b6-4fc8-ae81-94802cd42e09" (UID: "f10f3396-84b6-4fc8-ae81-94802cd42e09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:44.740261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.740209    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w" podUID="adfba4d2-eb18-484f-af0b-e4eb2771bb95" containerName="payload-processing" containerID="cri-o://7c3b15a6cab67ad84ad51d47bc44c9d41be43d7348ef2a89f31aa9bfd7df7a06" gracePeriod=25
Oct 30 21:11:44.756330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.754019    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt624\" (UniqueName: \"kubernetes.io/projected/f10f3396-84b6-4fc8-ae81-94802cd42e09-kube-api-access-zt624\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:44.756330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.754052    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f10f3396-84b6-4fc8-ae81-94802cd42e09-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:44.756330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.754065    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ea25b16-f21e-444d-a4fd-0faff56edf63-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:44.756330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.754080    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58256\" (UniqueName: \"kubernetes.io/projected/2ea25b16-f21e-444d-a4fd-0faff56edf63-kube-api-access-58256\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:44.948218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.948178    4918 generic.go:356] "Generic (PLEG): container finished" podID="2ea25b16-f21e-444d-a4fd-0faff56edf63" containerID="c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3" exitCode=0
Oct 30 21:11:44.948355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.948286    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts" event={"ID":"2ea25b16-f21e-444d-a4fd-0faff56edf63","Type":"ContainerDied","Data":"c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3"}
Oct 30 21:11:44.948429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.948369    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts" event={"ID":"2ea25b16-f21e-444d-a4fd-0faff56edf63","Type":"ContainerDied","Data":"f576c3a3197e011f3eb4a4e64d1088edef333c0c1c5b4566971601f086323951"}
Oct 30 21:11:44.948470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.948426    4918 scope.go:122] "RemoveContainer" containerID="c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3"
Oct 30 21:11:44.948588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.948574    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"
Oct 30 21:11:44.955316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.955287    4918 generic.go:356] "Generic (PLEG): container finished" podID="f10f3396-84b6-4fc8-ae81-94802cd42e09" containerID="e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48" exitCode=0
Oct 30 21:11:44.955509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.955483    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg" event={"ID":"f10f3396-84b6-4fc8-ae81-94802cd42e09","Type":"ContainerDied","Data":"e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48"}
Oct 30 21:11:44.955566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.955533    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg" event={"ID":"f10f3396-84b6-4fc8-ae81-94802cd42e09","Type":"ContainerDied","Data":"abf6301b593d2c473da0f4df990e1a84647bcba924e2600c7cc463ff1edf1982"}
Oct 30 21:11:44.955625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.955611    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"
Oct 30 21:11:44.969268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.969243    4918 generic.go:356] "Generic (PLEG): container finished" podID="adfba4d2-eb18-484f-af0b-e4eb2771bb95" containerID="7c3b15a6cab67ad84ad51d47bc44c9d41be43d7348ef2a89f31aa9bfd7df7a06" exitCode=0
Oct 30 21:11:44.969376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.969285    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w" event={"ID":"adfba4d2-eb18-484f-af0b-e4eb2771bb95","Type":"ContainerDied","Data":"7c3b15a6cab67ad84ad51d47bc44c9d41be43d7348ef2a89f31aa9bfd7df7a06"}
Oct 30 21:11:44.969580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.969428    4918 scope.go:122] "RemoveContainer" containerID="c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3"
Oct 30 21:11:44.970248 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:11:44.970086    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3\": container with ID starting with c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3 not found: ID does not exist" containerID="c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3"
Oct 30 21:11:44.970248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.970122    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3"} err="failed to get container status \"c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3\": rpc error: code = NotFound desc = could not find container \"c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3\": container with ID starting with c02d3355d2d6c4573c243e0650a7de3f9665b0120b57cdd29d9190dba1921da3 not found: ID does not exist"
Oct 30 21:11:44.970248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.970144    4918 scope.go:122] "RemoveContainer" containerID="e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48"
Oct 30 21:11:44.981088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.980975    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"]
Oct 30 21:11:44.984970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.984936    4918 scope.go:122] "RemoveContainer" containerID="e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48"
Oct 30 21:11:45.016462 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:11:44.999533    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48\": container with ID starting with e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48 not found: ID does not exist" containerID="e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48"
Oct 30 21:11:45.016462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:44.999576    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48"} err="failed to get container status \"e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48\": rpc error: code = NotFound desc = could not find container \"e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48\": container with ID starting with e1ba38018a91cd4229f63d5d6a7e3dc8c1c10021364074a2058bbd9bad74ff48 not found: ID does not exist"
Oct 30 21:11:45.024865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.024842    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f10f3396-84b6-4fc8-ae81-94802cd42e09" containerName="payload-processing"
Oct 30 21:11:45.024942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.024865    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10f3396-84b6-4fc8-ae81-94802cd42e09" containerName="payload-processing"
Oct 30 21:11:45.024942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.024902    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9f467c15-a3a0-44cd-81cf-52d80a253a58" containerName="revoke-keys"
Oct 30 21:11:45.024942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.024907    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f467c15-a3a0-44cd-81cf-52d80a253a58" containerName="revoke-keys"
Oct 30 21:11:45.024942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.024933    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2ea25b16-f21e-444d-a4fd-0faff56edf63" containerName="payload-pre-processing"
Oct 30 21:11:45.024942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.024938    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea25b16-f21e-444d-a4fd-0faff56edf63" containerName="payload-pre-processing"
Oct 30 21:11:45.025177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.025021    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f467c15-a3a0-44cd-81cf-52d80a253a58" containerName="revoke-keys"
Oct 30 21:11:45.025177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.025032    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea25b16-f21e-444d-a4fd-0faff56edf63" containerName="payload-pre-processing"
Oct 30 21:11:45.025177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.025040    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10f3396-84b6-4fc8-ae81-94802cd42e09" containerName="payload-processing"
Oct 30 21:11:45.031430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.029945    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"]
Oct 30 21:11:45.031430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.029992    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"]
Oct 30 21:11:45.031430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.030014    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e2cef583-6c8d7b99ff-fwzts"]
Oct 30 21:11:45.031430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.030090    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.033086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.033068    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"
Oct 30 21:11:45.041959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.041932    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"]
Oct 30 21:11:45.043629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.043606    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e2cef583-5b6cb675cc-wsrqg"]
Oct 30 21:11:45.079260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.079232    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea25b16-f21e-444d-a4fd-0faff56edf63" path="/var/lib/kubelet/pods/2ea25b16-f21e-444d-a4fd-0faff56edf63/volumes"
Oct 30 21:11:45.079619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.079605    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10f3396-84b6-4fc8-ae81-94802cd42e09" path="/var/lib/kubelet/pods/f10f3396-84b6-4fc8-ae81-94802cd42e09/volumes"
Oct 30 21:11:45.132428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.129749    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-384164-openshift-default-dockercfg-qs2n4\""
Oct 30 21:11:45.139965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.139931    4918 patch_prober.go:29] interesting pod/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 21:11:45.140108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.139994    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 21:11:45.159174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.159145    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfba4d2-eb18-484f-af0b-e4eb2771bb95-kube-api-access-s4bm5\" (UniqueName: \"kubernetes.io/projected/adfba4d2-eb18-484f-af0b-e4eb2771bb95-kube-api-access-s4bm5\") pod \"adfba4d2-eb18-484f-af0b-e4eb2771bb95\" (UID: \"adfba4d2-eb18-484f-af0b-e4eb2771bb95\") "
Oct 30 21:11:45.163856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.160995    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adfba4d2-eb18-484f-af0b-e4eb2771bb95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfba4d2-eb18-484f-af0b-e4eb2771bb95-plugins-config-volume\") pod \"adfba4d2-eb18-484f-af0b-e4eb2771bb95\" (UID: \"adfba4d2-eb18-484f-af0b-e4eb2771bb95\") "
Oct 30 21:11:45.163856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.161187    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.163856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.161249    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.163856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.161273    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.163856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.161349    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.163856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.161376    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxds5\" (UniqueName: \"kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-kube-api-access-qxds5\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.163856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.161563    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.163856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.161592    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.163856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.161619    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.163856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.161753    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.163856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.162202    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfba4d2-eb18-484f-af0b-e4eb2771bb95-plugins-config-volume" pod "adfba4d2-eb18-484f-af0b-e4eb2771bb95" (UID: "adfba4d2-eb18-484f-af0b-e4eb2771bb95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:45.164652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.164623    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfba4d2-eb18-484f-af0b-e4eb2771bb95-kube-api-access-s4bm5" pod "adfba4d2-eb18-484f-af0b-e4eb2771bb95" (UID: "adfba4d2-eb18-484f-af0b-e4eb2771bb95"). InnerVolumeSpecName "kube-api-access-s4bm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:45.263079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263049    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.263079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263078    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.263251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263114    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.263251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263142    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxds5\" (UniqueName: \"kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-kube-api-access-qxds5\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.263251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263240    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.263346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263263    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.263346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263291    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.263346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263336    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.263469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263405    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.263506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263481    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4bm5\" (UniqueName: \"kubernetes.io/projected/adfba4d2-eb18-484f-af0b-e4eb2771bb95-kube-api-access-s4bm5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:45.263506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263501    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adfba4d2-eb18-484f-af0b-e4eb2771bb95-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:45.263802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263756    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.264036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263827    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.264036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263854    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.264036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263873    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.264036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.263888    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.265681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.265660    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.265779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.265763    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.278512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.278461    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxds5\" (UniqueName: \"kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-kube-api-access-qxds5\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.278596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.278546    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-j97gk\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.347551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.347508    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:45.581002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.580924    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:45.659152 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.659130    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"]
Oct 30 21:11:45.660646 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:45.660617    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23c4448_ef53_4fe3_9b8d_6e69f5711f53.slice/crio-526af3da974c4f1081a80d742ade782ead96941c13bbc050348dd383dd64d599 WatchSource:0}: Error finding container 526af3da974c4f1081a80d742ade782ead96941c13bbc050348dd383dd64d599: Status 404 returned error can't find the container with id 526af3da974c4f1081a80d742ade782ead96941c13bbc050348dd383dd64d599
Oct 30 21:11:45.663164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.663134    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:45.663252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.663198    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:45.663252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.663234    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:45.979159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.979119    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" event={"ID":"f23c4448-ef53-4fe3-9b8d-6e69f5711f53","Type":"ContainerStarted","Data":"6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b"}
Oct 30 21:11:45.979368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.979217    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" event={"ID":"f23c4448-ef53-4fe3-9b8d-6e69f5711f53","Type":"ContainerStarted","Data":"526af3da974c4f1081a80d742ade782ead96941c13bbc050348dd383dd64d599"}
Oct 30 21:11:45.980647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.980612    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w" event={"ID":"adfba4d2-eb18-484f-af0b-e4eb2771bb95","Type":"ContainerDied","Data":"5f9b173007a39b1b62a84dec338b8dce344343e0793640f7bd2432b67b94bdc2"}
Oct 30 21:11:45.980647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.980625    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"
Oct 30 21:11:45.980813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:45.980658    4918 scope.go:122] "RemoveContainer" containerID="7c3b15a6cab67ad84ad51d47bc44c9d41be43d7348ef2a89f31aa9bfd7df7a06"
Oct 30 21:11:46.036433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.036239    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" podStartSLOduration=2.036218347 podStartE2EDuration="2.036218347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:46.025788841 +0000 UTC m=+619.072348326" watchObservedRunningTime="2026-10-30 21:11:46.036218347 +0000 UTC m=+619.082777499"
Oct 30 21:11:46.064679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.064649    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"]
Oct 30 21:11:46.071285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.071250    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6eedd82b-7fb4788d75-jfb7w"]
Oct 30 21:11:46.139173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.139140    4918 patch_prober.go:29] interesting pod/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 30 21:11:46.139344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.139192    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 30 21:11:46.348605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.348568    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:46.354580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.354534    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:46.380459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.379931    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"]
Oct 30 21:11:46.426074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.383690    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"]
Oct 30 21:11:46.518236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.518210    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:46.578450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578354    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istiod-ca-cert\") pod \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") "
Oct 30 21:11:46.578450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578421    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-envoy\") pod \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") "
Oct 30 21:11:46.578959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578523    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-socket\") pod \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") "
Oct 30 21:11:46.578959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578580    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-certs\") pod \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") "
Oct 30 21:11:46.578959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578641    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-credential-socket\") pod \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") "
Oct 30 21:11:46.578959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578719    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-podinfo\") pod \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") "
Oct 30 21:11:46.578959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578747    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-token\" (UniqueName: \"kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-token\") pod \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") "
Oct 30 21:11:46.578959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578753    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istiod-ca-cert" pod "1db5e191-2a8e-40d2-819c-e9c00f33f07e" (UID: "1db5e191-2a8e-40d2-819c-e9c00f33f07e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:46.578959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578753    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-socket" pod "1db5e191-2a8e-40d2-819c-e9c00f33f07e" (UID: "1db5e191-2a8e-40d2-819c-e9c00f33f07e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:46.578959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578840    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-kube-api-access-9bprq\" (UniqueName: \"kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-kube-api-access-9bprq\") pod \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") "
Oct 30 21:11:46.578959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578879    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-data\") pod \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\" (UID: \"1db5e191-2a8e-40d2-819c-e9c00f33f07e\") "
Oct 30 21:11:46.578959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578871    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-certs" pod "1db5e191-2a8e-40d2-819c-e9c00f33f07e" (UID: "1db5e191-2a8e-40d2-819c-e9c00f33f07e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:46.579491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.578973    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-credential-socket" pod "1db5e191-2a8e-40d2-819c-e9c00f33f07e" (UID: "1db5e191-2a8e-40d2-819c-e9c00f33f07e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:46.579491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.579226    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-data" pod "1db5e191-2a8e-40d2-819c-e9c00f33f07e" (UID: "1db5e191-2a8e-40d2-819c-e9c00f33f07e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:46.579491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.579238    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:46.579491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.579257    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:46.579491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.579266    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:46.579491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.579276    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:46.580824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.580801    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-envoy" pod "1db5e191-2a8e-40d2-819c-e9c00f33f07e" (UID: "1db5e191-2a8e-40d2-819c-e9c00f33f07e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:46.581156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.581134    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-kube-api-access-9bprq" pod "1db5e191-2a8e-40d2-819c-e9c00f33f07e" (UID: "1db5e191-2a8e-40d2-819c-e9c00f33f07e"). InnerVolumeSpecName "kube-api-access-9bprq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:46.581438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.581418    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-podinfo" pod "1db5e191-2a8e-40d2-819c-e9c00f33f07e" (UID: "1db5e191-2a8e-40d2-819c-e9c00f33f07e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:11:46.581515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.581440    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-token" pod "1db5e191-2a8e-40d2-819c-e9c00f33f07e" (UID: "1db5e191-2a8e-40d2-819c-e9c00f33f07e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:46.680773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.680741    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:46.680773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.680772    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:46.680975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.680783    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bprq\" (UniqueName: \"kubernetes.io/projected/1db5e191-2a8e-40d2-819c-e9c00f33f07e-kube-api-access-9bprq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:46.680975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.680793    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:46.680975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.680803    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1db5e191-2a8e-40d2-819c-e9c00f33f07e-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:46.989641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.989199    4918 generic.go:356] "Generic (PLEG): container finished" podID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerID="ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0" exitCode=0
Oct 30 21:11:46.989641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.989302    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" event={"ID":"1db5e191-2a8e-40d2-819c-e9c00f33f07e","Type":"ContainerDied","Data":"ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0"}
Oct 30 21:11:46.989641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.989323    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"
Oct 30 21:11:46.989641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.989361    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x" event={"ID":"1db5e191-2a8e-40d2-819c-e9c00f33f07e","Type":"ContainerDied","Data":"3840a5d3d195e2ed32e7f3282cf61da7b019e27728c6ee79ad3385a60ffbcfc3"}
Oct 30 21:11:46.989641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.989426    4918 scope.go:122] "RemoveContainer" containerID="ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0"
Oct 30 21:11:46.991501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.991314    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:46.992969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:46.992946    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:11:47.002836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.002818    4918 scope.go:122] "RemoveContainer" containerID="7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9"
Oct 30 21:11:47.013979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.013944    4918 scope.go:122] "RemoveContainer" containerID="ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0"
Oct 30 21:11:47.014325 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:11:47.014299    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0\": container with ID starting with ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0 not found: ID does not exist" containerID="ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0"
Oct 30 21:11:47.014422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.014339    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0"} err="failed to get container status \"ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0\": rpc error: code = NotFound desc = could not find container \"ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0\": container with ID starting with ff51ef22ba4c020f1039199b76190b8138475035ca4330845e6aeb1a3147d8e0 not found: ID does not exist"
Oct 30 21:11:47.014422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.014361    4918 scope.go:122] "RemoveContainer" containerID="7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9"
Oct 30 21:11:47.014731 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:11:47.014709    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9\": container with ID starting with 7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9 not found: ID does not exist" containerID="7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9"
Oct 30 21:11:47.014792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.014752    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9"} err="failed to get container status \"7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9\": rpc error: code = NotFound desc = could not find container \"7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9\": container with ID starting with 7ee32f908a42f187f6cc4ad2045e7b23e41c7f6ec695a5005493650216f90ea9 not found: ID does not exist"
Oct 30 21:11:47.055970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.055935    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"]
Oct 30 21:11:47.058982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.058953    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e2cef583-openshift-default-57d976f46c-5kq4x"]
Oct 30 21:11:47.078977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.078949    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" path="/var/lib/kubelet/pods/1db5e191-2a8e-40d2-819c-e9c00f33f07e/volumes"
Oct 30 21:11:47.079414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.079380    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adfba4d2-eb18-484f-af0b-e4eb2771bb95" path="/var/lib/kubelet/pods/adfba4d2-eb18-484f-af0b-e4eb2771bb95/volumes"
Oct 30 21:11:47.390712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.390624    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:47.392026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.392005    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:47.574036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.574008    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"]
Oct 30 21:11:47.579952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.579927    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5ee9aa6f-qs54t"]
Oct 30 21:11:47.808801 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.808773    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"]
Oct 30 21:11:47.999478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:47.999448    4918 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" secret="" err="secret \"e2e-derive-5ee9aa6f-openshift-default-dockercfg-2c4x8\" not found"
Oct 30 21:11:48.175765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.175691    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"]
Oct 30 21:11:48.176087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.176076    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy"
Oct 30 21:11:48.176126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.176089    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy"
Oct 30 21:11:48.176126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.176098    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy"
Oct 30 21:11:48.176126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.176103    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy"
Oct 30 21:11:48.176211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.176128    4918 cpu_manager.go:414] "containerMap: removing container" podUID="adfba4d2-eb18-484f-af0b-e4eb2771bb95" containerName="payload-processing"
Oct 30 21:11:48.176211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.176134    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adfba4d2-eb18-484f-af0b-e4eb2771bb95" containerName="payload-processing"
Oct 30 21:11:48.176211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.176194    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy"
Oct 30 21:11:48.176211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.176202    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db5e191-2a8e-40d2-819c-e9c00f33f07e" containerName="istio-proxy"
Oct 30 21:11:48.176211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.176210    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="adfba4d2-eb18-484f-af0b-e4eb2771bb95" containerName="payload-processing"
Oct 30 21:11:48.180719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.180700    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"
Oct 30 21:11:48.188897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.188562    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-bfc50e-dockercfg-v6plk\""
Oct 30 21:11:48.189923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.189896    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"]
Oct 30 21:11:48.192910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.192890    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-bfc50e\""
Oct 30 21:11:48.198444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.198286    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98af039-b120-4448-a7d9-a9e4a00ef226-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs\" (UID: \"c98af039-b120-4448-a7d9-a9e4a00ef226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"
Oct 30 21:11:48.198444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.198377    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wg47\" (UniqueName: \"kubernetes.io/projected/c98af039-b120-4448-a7d9-a9e4a00ef226-kube-api-access-6wg47\") pod \"payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs\" (UID: \"c98af039-b120-4448-a7d9-a9e4a00ef226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"
Oct 30 21:11:48.199903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.199884    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"]
Oct 30 21:11:48.203623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.203605    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"
Oct 30 21:11:48.213159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.213136    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"]
Oct 30 21:11:48.299666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.299627    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98af039-b120-4448-a7d9-a9e4a00ef226-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs\" (UID: \"c98af039-b120-4448-a7d9-a9e4a00ef226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"
Oct 30 21:11:48.299868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.299715    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wg47\" (UniqueName: \"kubernetes.io/projected/c98af039-b120-4448-a7d9-a9e4a00ef226-kube-api-access-6wg47\") pod \"payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs\" (UID: \"c98af039-b120-4448-a7d9-a9e4a00ef226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"
Oct 30 21:11:48.299868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.299842    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsqh6\" (UniqueName: \"kubernetes.io/projected/49552f5c-97a2-4efb-b7e7-674879eb98cf-kube-api-access-hsqh6\") pod \"payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb\" (UID: \"49552f5c-97a2-4efb-b7e7-674879eb98cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"
Oct 30 21:11:48.300176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.300061    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49552f5c-97a2-4efb-b7e7-674879eb98cf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb\" (UID: \"49552f5c-97a2-4efb-b7e7-674879eb98cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"
Oct 30 21:11:48.300810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.300780    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98af039-b120-4448-a7d9-a9e4a00ef226-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs\" (UID: \"c98af039-b120-4448-a7d9-a9e4a00ef226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"
Oct 30 21:11:48.315812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.315783    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wg47\" (UniqueName: \"kubernetes.io/projected/c98af039-b120-4448-a7d9-a9e4a00ef226-kube-api-access-6wg47\") pod \"payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs\" (UID: \"c98af039-b120-4448-a7d9-a9e4a00ef226\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"
Oct 30 21:11:48.400792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.400758    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsqh6\" (UniqueName: \"kubernetes.io/projected/49552f5c-97a2-4efb-b7e7-674879eb98cf-kube-api-access-hsqh6\") pod \"payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb\" (UID: \"49552f5c-97a2-4efb-b7e7-674879eb98cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"
Oct 30 21:11:48.401185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.400816    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49552f5c-97a2-4efb-b7e7-674879eb98cf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb\" (UID: \"49552f5c-97a2-4efb-b7e7-674879eb98cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"
Oct 30 21:11:48.401552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.401536    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49552f5c-97a2-4efb-b7e7-674879eb98cf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb\" (UID: \"49552f5c-97a2-4efb-b7e7-674879eb98cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"
Oct 30 21:11:48.410732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.410712    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsqh6\" (UniqueName: \"kubernetes.io/projected/49552f5c-97a2-4efb-b7e7-674879eb98cf-kube-api-access-hsqh6\") pod \"payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb\" (UID: \"49552f5c-97a2-4efb-b7e7-674879eb98cf\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"
Oct 30 21:11:48.503673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.503643    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"
Oct 30 21:11:48.521907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.521881    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"
Oct 30 21:11:48.795165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.795137    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"]
Oct 30 21:11:48.796475 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:48.796446    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98af039_b120_4448_a7d9_a9e4a00ef226.slice/crio-84d56f956fcc7ce3868de5b25b54b97fb9e86115389ee89f38e29c96f29a3eb6 WatchSource:0}: Error finding container 84d56f956fcc7ce3868de5b25b54b97fb9e86115389ee89f38e29c96f29a3eb6: Status 404 returned error can't find the container with id 84d56f956fcc7ce3868de5b25b54b97fb9e86115389ee89f38e29c96f29a3eb6
Oct 30 21:11:48.823501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.823478    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"]
Oct 30 21:11:48.829893 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:48.829823    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49552f5c_97a2_4efb_b7e7_674879eb98cf.slice/crio-33363e1aa5f4c57e717857948d8456ec92aa4e3a4aca36ee43d20a24a74310f5 WatchSource:0}: Error finding container 33363e1aa5f4c57e717857948d8456ec92aa4e3a4aca36ee43d20a24a74310f5: Status 404 returned error can't find the container with id 33363e1aa5f4c57e717857948d8456ec92aa4e3a4aca36ee43d20a24a74310f5
Oct 30 21:11:48.916783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:48.916762    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:49.006907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.006875    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-credential-socket\") pod \"f24d688c-575f-4c80-8745-8cc0d72ad30c\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") "
Oct 30 21:11:49.007434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.006923    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f24d688c-575f-4c80-8745-8cc0d72ad30c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24d688c-575f-4c80-8745-8cc0d72ad30c-istiod-ca-cert\") pod \"f24d688c-575f-4c80-8745-8cc0d72ad30c\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") "
Oct 30 21:11:49.007434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.006988    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-podinfo\") pod \"f24d688c-575f-4c80-8745-8cc0d72ad30c\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") "
Oct 30 21:11:49.007434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.007016    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-data\") pod \"f24d688c-575f-4c80-8745-8cc0d72ad30c\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") "
Oct 30 21:11:49.007434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.007052    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-certs\") pod \"f24d688c-575f-4c80-8745-8cc0d72ad30c\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") "
Oct 30 21:11:49.007434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.007098    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-kube-api-access-9f5cf\" (UniqueName: \"kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-kube-api-access-9f5cf\") pod \"f24d688c-575f-4c80-8745-8cc0d72ad30c\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") "
Oct 30 21:11:49.007434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.007126    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-credential-socket" pod "f24d688c-575f-4c80-8745-8cc0d72ad30c" (UID: "f24d688c-575f-4c80-8745-8cc0d72ad30c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:49.007434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.007205    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-token\" (UniqueName: \"kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-token\") pod \"f24d688c-575f-4c80-8745-8cc0d72ad30c\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") "
Oct 30 21:11:49.007434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.007233    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-envoy\") pod \"f24d688c-575f-4c80-8745-8cc0d72ad30c\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") "
Oct 30 21:11:49.007434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.007312    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-socket\") pod \"f24d688c-575f-4c80-8745-8cc0d72ad30c\" (UID: \"f24d688c-575f-4c80-8745-8cc0d72ad30c\") "
Oct 30 21:11:49.008363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.007739    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:49.008363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.007973    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f24d688c-575f-4c80-8745-8cc0d72ad30c-istiod-ca-cert" pod "f24d688c-575f-4c80-8745-8cc0d72ad30c" (UID: "f24d688c-575f-4c80-8745-8cc0d72ad30c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:49.008363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.008334    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-certs" pod "f24d688c-575f-4c80-8745-8cc0d72ad30c" (UID: "f24d688c-575f-4c80-8745-8cc0d72ad30c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:49.013327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.008475    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-data" pod "f24d688c-575f-4c80-8745-8cc0d72ad30c" (UID: "f24d688c-575f-4c80-8745-8cc0d72ad30c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:49.013327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.010954    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb" event={"ID":"49552f5c-97a2-4efb-b7e7-674879eb98cf","Type":"ContainerStarted","Data":"eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8"}
Oct 30 21:11:49.013327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.011013    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb" event={"ID":"49552f5c-97a2-4efb-b7e7-674879eb98cf","Type":"ContainerStarted","Data":"33363e1aa5f4c57e717857948d8456ec92aa4e3a4aca36ee43d20a24a74310f5"}
Oct 30 21:11:49.013327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.011595    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"
Oct 30 21:11:49.013327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.012139    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-socket" pod "f24d688c-575f-4c80-8745-8cc0d72ad30c" (UID: "f24d688c-575f-4c80-8745-8cc0d72ad30c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:49.016148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.016125    4918 generic.go:356] "Generic (PLEG): container finished" podID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerID="6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df" exitCode=0
Oct 30 21:11:49.016372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.016303    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"
Oct 30 21:11:49.016372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.016334    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" event={"ID":"f24d688c-575f-4c80-8745-8cc0d72ad30c","Type":"ContainerDied","Data":"6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df"}
Oct 30 21:11:49.016372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.016339    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-podinfo" pod "f24d688c-575f-4c80-8745-8cc0d72ad30c" (UID: "f24d688c-575f-4c80-8745-8cc0d72ad30c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:11:49.016567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.016381    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-kube-api-access-9f5cf" pod "f24d688c-575f-4c80-8745-8cc0d72ad30c" (UID: "f24d688c-575f-4c80-8745-8cc0d72ad30c"). InnerVolumeSpecName "kube-api-access-9f5cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:49.016567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.016412    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx" event={"ID":"f24d688c-575f-4c80-8745-8cc0d72ad30c","Type":"ContainerDied","Data":"f6fc00e5d746352482a506ba42e8335bb51fd7a78a783376de01a6b5f8333b4e"}
Oct 30 21:11:49.016567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.016382    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-envoy" pod "f24d688c-575f-4c80-8745-8cc0d72ad30c" (UID: "f24d688c-575f-4c80-8745-8cc0d72ad30c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:49.016567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.016443    4918 scope.go:122] "RemoveContainer" containerID="6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df"
Oct 30 21:11:49.018171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.018151    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-token" pod "f24d688c-575f-4c80-8745-8cc0d72ad30c" (UID: "f24d688c-575f-4c80-8745-8cc0d72ad30c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:49.018752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.018725    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs" event={"ID":"c98af039-b120-4448-a7d9-a9e4a00ef226","Type":"ContainerStarted","Data":"166bc2dce5cb3cb357bd4566b40857caf66c6524f95beae268835119286a0be0"}
Oct 30 21:11:49.018852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.018780    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs" event={"ID":"c98af039-b120-4448-a7d9-a9e4a00ef226","Type":"ContainerStarted","Data":"84d56f956fcc7ce3868de5b25b54b97fb9e86115389ee89f38e29c96f29a3eb6"}
Oct 30 21:11:49.018962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.018947    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"
Oct 30 21:11:49.019026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.018973    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerName="istio-proxy" containerID="cri-o://8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30" gracePeriod=30
Oct 30 21:11:49.026778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.026760    4918 scope.go:122] "RemoveContainer" containerID="0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4"
Oct 30 21:11:49.035881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.035852    4918 scope.go:122] "RemoveContainer" containerID="6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df"
Oct 30 21:11:49.036165 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:11:49.036135    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df\": container with ID starting with 6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df not found: ID does not exist" containerID="6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df"
Oct 30 21:11:49.036209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.036178    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df"} err="failed to get container status \"6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df\": rpc error: code = NotFound desc = could not find container \"6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df\": container with ID starting with 6ac61a7b5fd33ae6cceedcfb66605f07f969a38c286bd50b8f76bbc1f79898df not found: ID does not exist"
Oct 30 21:11:49.036209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.036196    4918 scope.go:122] "RemoveContainer" containerID="0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4"
Oct 30 21:11:49.036481 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:11:49.036463    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4\": container with ID starting with 0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4 not found: ID does not exist" containerID="0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4"
Oct 30 21:11:49.036548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.036490    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4"} err="failed to get container status \"0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4\": rpc error: code = NotFound desc = could not find container \"0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4\": container with ID starting with 0b47bf0c75fc4c7e6ca59643cbe666dbcb13d184efc02eb8a7f068bdf5fb68f4 not found: ID does not exist"
Oct 30 21:11:49.036764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.036730    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb" podStartSLOduration=1.036718132 podStartE2EDuration="1.036718132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:49.035184049 +0000 UTC m=+622.081743197" watchObservedRunningTime="2026-10-30 21:11:49.036718132 +0000 UTC m=+622.083277319"
Oct 30 21:11:49.076721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.076697    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f467c15-a3a0-44cd-81cf-52d80a253a58" path="/var/lib/kubelet/pods/9f467c15-a3a0-44cd-81cf-52d80a253a58/volumes"
Oct 30 21:11:49.108981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.108951    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:49.109143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.108985    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:49.109143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.109001    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:49.109143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.109015    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f24d688c-575f-4c80-8745-8cc0d72ad30c-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:49.109143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.109028    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:49.109143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.109041    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:49.109143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.109055    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f24d688c-575f-4c80-8745-8cc0d72ad30c-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:49.109143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.109070    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f5cf\" (UniqueName: \"kubernetes.io/projected/f24d688c-575f-4c80-8745-8cc0d72ad30c-kube-api-access-9f5cf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:49.119649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.119585    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs" podStartSLOduration=1.119571646 podStartE2EDuration="1.119571646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:49.093543829 +0000 UTC m=+622.140102978" watchObservedRunningTime="2026-10-30 21:11:49.119571646 +0000 UTC m=+622.166130790"
Oct 30 21:11:49.120146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.120122    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"]
Oct 30 21:11:49.120610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.120595    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerName="istio-proxy"
Oct 30 21:11:49.120664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.120612    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerName="istio-proxy"
Oct 30 21:11:49.120664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.120638    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerName="istio-proxy"
Oct 30 21:11:49.120664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.120644    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerName="istio-proxy"
Oct 30 21:11:49.120762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.120697    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerName="istio-proxy"
Oct 30 21:11:49.125065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.125048    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.141539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.141508    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a33b12-openshift-default-dockercfg-6bm8g\""
Oct 30 21:11:49.142879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.142822    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"]
Oct 30 21:11:49.210564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.210528    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.210808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.210601    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.210808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.210637    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.210808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.210689    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.210808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.210714    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.210808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.210743    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.211170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.210822    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc69c\" (UniqueName: \"kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-kube-api-access-vc69c\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.211170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.210850    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.211170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.210885    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c84dfa1-93db-4289-85a3-0182e98b41cf-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.311845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.311752    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.311845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.311796    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.311845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.311829    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.312275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.311913    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc69c\" (UniqueName: \"kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-kube-api-access-vc69c\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.312275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.311949    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.312275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.311987    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c84dfa1-93db-4289-85a3-0182e98b41cf-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.312275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.312048    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.312275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.312104    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.312275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.312133    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.312631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.312342    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.313030 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.313003    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.313159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.313094    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c84dfa1-93db-4289-85a3-0182e98b41cf-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.313242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.313226    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.313294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.313271    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.314556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.314537    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.314856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.314834    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.327737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.327714    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.327889 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.327872    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc69c\" (UniqueName: \"kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-kube-api-access-vc69c\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.385872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.385840    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"]
Oct 30 21:11:49.390580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.390555    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-111fd691-openshift-default-598486d85f-njfwx"]
Oct 30 21:11:49.437595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.437564    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:49.725680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.725647    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"]
Oct 30 21:11:49.728355 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:49.728327    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c84dfa1_93db_4289_85a3_0182e98b41cf.slice/crio-6e7ee2c22b5dc489d6ab152052e18d8e46a006c4e458916e7e340182b4a2cc82 WatchSource:0}: Error finding container 6e7ee2c22b5dc489d6ab152052e18d8e46a006c4e458916e7e340182b4a2cc82: Status 404 returned error can't find the container with id 6e7ee2c22b5dc489d6ab152052e18d8e46a006c4e458916e7e340182b4a2cc82
Oct 30 21:11:49.731471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.731440    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:49.731585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.731522    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:49.731585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:49.731570    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:11:50.026936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.026843    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" event={"ID":"0c84dfa1-93db-4289-85a3-0182e98b41cf","Type":"ContainerStarted","Data":"54b994aa348f3ddd9bc2ba22b8ec7752ed568f0245513d43d7f990f49fd89d5a"}
Oct 30 21:11:50.026936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.026908    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" event={"ID":"0c84dfa1-93db-4289-85a3-0182e98b41cf","Type":"ContainerStarted","Data":"6e7ee2c22b5dc489d6ab152052e18d8e46a006c4e458916e7e340182b4a2cc82"}
Oct 30 21:11:50.049572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.049515    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" podStartSLOduration=1.04949975 podStartE2EDuration="1.04949975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:50.047804351 +0000 UTC m=+623.094363540" watchObservedRunningTime="2026-10-30 21:11:50.04949975 +0000 UTC m=+623.096058895"
Oct 30 21:11:50.437951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.437868    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:50.443348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.443327    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:50.763897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.763861    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"]
Oct 30 21:11:50.764338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.764325    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" containerName="istio-proxy"
Oct 30 21:11:50.767485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.767471    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"
Oct 30 21:11:50.797092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.797061    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"]
Oct 30 21:11:50.803764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.803736    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-384164-dockercfg-lwrz9\""
Oct 30 21:11:50.803940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.803807    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-384164\""
Oct 30 21:11:50.847803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.845067    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036bea86-70eb-48e1-b108-b71928175a68-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g\" (UID: \"036bea86-70eb-48e1-b108-b71928175a68\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"
Oct 30 21:11:50.848124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.848103    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92xg\" (UniqueName: \"kubernetes.io/projected/036bea86-70eb-48e1-b108-b71928175a68-kube-api-access-j92xg\") pod \"payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g\" (UID: \"036bea86-70eb-48e1-b108-b71928175a68\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"
Oct 30 21:11:50.948795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.948762    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036bea86-70eb-48e1-b108-b71928175a68-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g\" (UID: \"036bea86-70eb-48e1-b108-b71928175a68\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"
Oct 30 21:11:50.948944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.948807    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92xg\" (UniqueName: \"kubernetes.io/projected/036bea86-70eb-48e1-b108-b71928175a68-kube-api-access-j92xg\") pod \"payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g\" (UID: \"036bea86-70eb-48e1-b108-b71928175a68\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"
Oct 30 21:11:50.949454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.949434    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036bea86-70eb-48e1-b108-b71928175a68-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g\" (UID: \"036bea86-70eb-48e1-b108-b71928175a68\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"
Oct 30 21:11:50.961544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:50.961511    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92xg\" (UniqueName: \"kubernetes.io/projected/036bea86-70eb-48e1-b108-b71928175a68-kube-api-access-j92xg\") pod \"payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g\" (UID: \"036bea86-70eb-48e1-b108-b71928175a68\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"
Oct 30 21:11:51.033179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.033100    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:51.034092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.034073    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:11:51.078278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.078242    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f24d688c-575f-4c80-8745-8cc0d72ad30c" path="/var/lib/kubelet/pods/f24d688c-575f-4c80-8745-8cc0d72ad30c/volumes"
Oct 30 21:11:51.095179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.095152    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"
Oct 30 21:11:51.375334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.375303    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"]
Oct 30 21:11:51.377111 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:51.377082    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036bea86_70eb_48e1_b108_b71928175a68.slice/crio-a84cb72fbd5f6f8c0ee0fa2ecadf0e5c5fa3ae71d6a22f6da8502582346759d4 WatchSource:0}: Error finding container a84cb72fbd5f6f8c0ee0fa2ecadf0e5c5fa3ae71d6a22f6da8502582346759d4: Status 404 returned error can't find the container with id a84cb72fbd5f6f8c0ee0fa2ecadf0e5c5fa3ae71d6a22f6da8502582346759d4
Oct 30 21:11:51.380862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.380727    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889" podUID="11880667-21f9-4cb0-973a-1a24773fd070" containerName="payload-processing" containerID="cri-o://8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e" gracePeriod=25
Oct 30 21:11:51.384274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.384227    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn" podUID="e09b1342-195a-401d-a1bb-2827378fb501" containerName="payload-pre-processing" containerID="cri-o://16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee" gracePeriod=25
Oct 30 21:11:51.699628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.699604    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"
Oct 30 21:11:51.702204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.702188    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"
Oct 30 21:11:51.760731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.760702    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e09b1342-195a-401d-a1bb-2827378fb501-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09b1342-195a-401d-a1bb-2827378fb501-plugins-config-volume\") pod \"e09b1342-195a-401d-a1bb-2827378fb501\" (UID: \"e09b1342-195a-401d-a1bb-2827378fb501\") "
Oct 30 21:11:51.760895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.760795    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e09b1342-195a-401d-a1bb-2827378fb501-kube-api-access-9xzjn\" (UniqueName: \"kubernetes.io/projected/e09b1342-195a-401d-a1bb-2827378fb501-kube-api-access-9xzjn\") pod \"e09b1342-195a-401d-a1bb-2827378fb501\" (UID: \"e09b1342-195a-401d-a1bb-2827378fb501\") "
Oct 30 21:11:51.760895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.760874    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11880667-21f9-4cb0-973a-1a24773fd070-kube-api-access-fkdj7\" (UniqueName: \"kubernetes.io/projected/11880667-21f9-4cb0-973a-1a24773fd070-kube-api-access-fkdj7\") pod \"11880667-21f9-4cb0-973a-1a24773fd070\" (UID: \"11880667-21f9-4cb0-973a-1a24773fd070\") "
Oct 30 21:11:51.761024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.760899    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11880667-21f9-4cb0-973a-1a24773fd070-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11880667-21f9-4cb0-973a-1a24773fd070-plugins-config-volume\") pod \"11880667-21f9-4cb0-973a-1a24773fd070\" (UID: \"11880667-21f9-4cb0-973a-1a24773fd070\") "
Oct 30 21:11:51.761088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.761062    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09b1342-195a-401d-a1bb-2827378fb501-plugins-config-volume" pod "e09b1342-195a-401d-a1bb-2827378fb501" (UID: "e09b1342-195a-401d-a1bb-2827378fb501"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:51.761296 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.761280    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09b1342-195a-401d-a1bb-2827378fb501-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:51.761355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.761317    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11880667-21f9-4cb0-973a-1a24773fd070-plugins-config-volume" pod "11880667-21f9-4cb0-973a-1a24773fd070" (UID: "11880667-21f9-4cb0-973a-1a24773fd070"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:51.763128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.763103    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09b1342-195a-401d-a1bb-2827378fb501-kube-api-access-9xzjn" pod "e09b1342-195a-401d-a1bb-2827378fb501" (UID: "e09b1342-195a-401d-a1bb-2827378fb501"). InnerVolumeSpecName "kube-api-access-9xzjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:51.763227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.763129    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11880667-21f9-4cb0-973a-1a24773fd070-kube-api-access-fkdj7" pod "11880667-21f9-4cb0-973a-1a24773fd070" (UID: "11880667-21f9-4cb0-973a-1a24773fd070"). InnerVolumeSpecName "kube-api-access-fkdj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:51.862169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.862140    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xzjn\" (UniqueName: \"kubernetes.io/projected/e09b1342-195a-401d-a1bb-2827378fb501-kube-api-access-9xzjn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:51.862169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.862165    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkdj7\" (UniqueName: \"kubernetes.io/projected/11880667-21f9-4cb0-973a-1a24773fd070-kube-api-access-fkdj7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:51.862169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:51.862174    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11880667-21f9-4cb0-973a-1a24773fd070-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:52.039500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.039465    4918 generic.go:356] "Generic (PLEG): container finished" podID="e09b1342-195a-401d-a1bb-2827378fb501" containerID="16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee" exitCode=0
Oct 30 21:11:52.039683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.039525    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn" event={"ID":"e09b1342-195a-401d-a1bb-2827378fb501","Type":"ContainerDied","Data":"16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee"}
Oct 30 21:11:52.039683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.039582    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn" event={"ID":"e09b1342-195a-401d-a1bb-2827378fb501","Type":"ContainerDied","Data":"630f39b9c0fc27252856e1a3016d1c08e7adcd3d95184e69d53bd62aff77218a"}
Oct 30 21:11:52.039683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.039533    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"
Oct 30 21:11:52.039683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.039603    4918 scope.go:122] "RemoveContainer" containerID="16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee"
Oct 30 21:11:52.041538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.041518    4918 generic.go:356] "Generic (PLEG): container finished" podID="11880667-21f9-4cb0-973a-1a24773fd070" containerID="8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e" exitCode=0
Oct 30 21:11:52.041669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.041572    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"
Oct 30 21:11:52.041669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.041584    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889" event={"ID":"11880667-21f9-4cb0-973a-1a24773fd070","Type":"ContainerDied","Data":"8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e"}
Oct 30 21:11:52.041669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.041624    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889" event={"ID":"11880667-21f9-4cb0-973a-1a24773fd070","Type":"ContainerDied","Data":"80cb55a9f689d7eb63d8cd6c12e2889fb2de2a7e8923ceebfebcd47e4a24f390"}
Oct 30 21:11:52.043797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.043263    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g" event={"ID":"036bea86-70eb-48e1-b108-b71928175a68","Type":"ContainerStarted","Data":"799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb"}
Oct 30 21:11:52.043797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.043315    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g" event={"ID":"036bea86-70eb-48e1-b108-b71928175a68","Type":"ContainerStarted","Data":"a84cb72fbd5f6f8c0ee0fa2ecadf0e5c5fa3ae71d6a22f6da8502582346759d4"}
Oct 30 21:11:52.050345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.050329    4918 scope.go:122] "RemoveContainer" containerID="16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee"
Oct 30 21:11:52.050634 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:11:52.050615    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee\": container with ID starting with 16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee not found: ID does not exist" containerID="16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee"
Oct 30 21:11:52.050699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.050641    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee"} err="failed to get container status \"16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee\": rpc error: code = NotFound desc = could not find container \"16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee\": container with ID starting with 16458ed6e805efb20c250f6c48640a45be90c1cf6b2c034ed01adb3e7f2050ee not found: ID does not exist"
Oct 30 21:11:52.050699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.050657    4918 scope.go:122] "RemoveContainer" containerID="8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e"
Oct 30 21:11:52.058987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.058971    4918 scope.go:122] "RemoveContainer" containerID="8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e"
Oct 30 21:11:52.059246 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:11:52.059230    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e\": container with ID starting with 8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e not found: ID does not exist" containerID="8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e"
Oct 30 21:11:52.059290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.059253    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e"} err="failed to get container status \"8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e\": rpc error: code = NotFound desc = could not find container \"8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e\": container with ID starting with 8beb47809608b0dd2bf244b9967bb392d55c45226a16f164629fcc22cee7874e not found: ID does not exist"
Oct 30 21:11:52.064920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.064885    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g" podStartSLOduration=2.064874637 podStartE2EDuration="2.064874637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:52.063080076 +0000 UTC m=+625.109639240" watchObservedRunningTime="2026-10-30 21:11:52.064874637 +0000 UTC m=+625.111433782"
Oct 30 21:11:52.081807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.081789    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"]
Oct 30 21:11:52.085660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.085638    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-5ee9aa6f-8496fc6b67-cwmwn"]
Oct 30 21:11:52.112067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.112041    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"]
Oct 30 21:11:52.118663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:52.118642    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-5ee9aa6f-56cffb98f5-nl889"]
Oct 30 21:11:53.050372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.050336    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"
Oct 30 21:11:53.075654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.075625    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11880667-21f9-4cb0-973a-1a24773fd070" path="/var/lib/kubelet/pods/11880667-21f9-4cb0-973a-1a24773fd070/volumes"
Oct 30 21:11:53.075986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.075974    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e09b1342-195a-401d-a1bb-2827378fb501" path="/var/lib/kubelet/pods/e09b1342-195a-401d-a1bb-2827378fb501/volumes"
Oct 30 21:11:53.858280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.858250    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"]
Oct 30 21:11:53.858743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.858729    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e09b1342-195a-401d-a1bb-2827378fb501" containerName="payload-pre-processing"
Oct 30 21:11:53.858800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.858746    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e09b1342-195a-401d-a1bb-2827378fb501" containerName="payload-pre-processing"
Oct 30 21:11:53.858800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.858756    4918 cpu_manager.go:414] "containerMap: removing container" podUID="11880667-21f9-4cb0-973a-1a24773fd070" containerName="payload-processing"
Oct 30 21:11:53.858800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.858762    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11880667-21f9-4cb0-973a-1a24773fd070" containerName="payload-processing"
Oct 30 21:11:53.858931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.858866    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e09b1342-195a-401d-a1bb-2827378fb501" containerName="payload-pre-processing"
Oct 30 21:11:53.858931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.858876    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="11880667-21f9-4cb0-973a-1a24773fd070" containerName="payload-processing"
Oct 30 21:11:53.863372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.863356    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"
Oct 30 21:11:53.872512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.872490    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a33b12-dockercfg-m8pfb\""
Oct 30 21:11:53.872653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.872538    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a33b12\""
Oct 30 21:11:53.874646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.874624    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"]
Oct 30 21:11:53.903342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.903308    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"]
Oct 30 21:11:53.906808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.906792    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"
Oct 30 21:11:53.921975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.921948    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"]
Oct 30 21:11:53.991233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.991194    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13a507e-48ca-4b4a-90dd-72f30316c75d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7\" (UID: \"a13a507e-48ca-4b4a-90dd-72f30316c75d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"
Oct 30 21:11:53.991660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.991638    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ndjj\" (UniqueName: \"kubernetes.io/projected/ff7f4bf0-e541-4893-a000-027b14e42b74-kube-api-access-5ndjj\") pod \"payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm\" (UID: \"ff7f4bf0-e541-4893-a000-027b14e42b74\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"
Oct 30 21:11:53.991955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.991907    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnbtg\" (UniqueName: \"kubernetes.io/projected/a13a507e-48ca-4b4a-90dd-72f30316c75d-kube-api-access-hnbtg\") pod \"payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7\" (UID: \"a13a507e-48ca-4b4a-90dd-72f30316c75d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"
Oct 30 21:11:53.992226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:53.992113    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7f4bf0-e541-4893-a000-027b14e42b74-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm\" (UID: \"ff7f4bf0-e541-4893-a000-027b14e42b74\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"
Oct 30 21:11:54.093683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.093654    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnbtg\" (UniqueName: \"kubernetes.io/projected/a13a507e-48ca-4b4a-90dd-72f30316c75d-kube-api-access-hnbtg\") pod \"payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7\" (UID: \"a13a507e-48ca-4b4a-90dd-72f30316c75d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"
Oct 30 21:11:54.094161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.093691    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7f4bf0-e541-4893-a000-027b14e42b74-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm\" (UID: \"ff7f4bf0-e541-4893-a000-027b14e42b74\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"
Oct 30 21:11:54.094161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.093748    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13a507e-48ca-4b4a-90dd-72f30316c75d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7\" (UID: \"a13a507e-48ca-4b4a-90dd-72f30316c75d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"
Oct 30 21:11:54.094161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.093779    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndjj\" (UniqueName: \"kubernetes.io/projected/ff7f4bf0-e541-4893-a000-027b14e42b74-kube-api-access-5ndjj\") pod \"payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm\" (UID: \"ff7f4bf0-e541-4893-a000-027b14e42b74\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"
Oct 30 21:11:54.094610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.094590    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7f4bf0-e541-4893-a000-027b14e42b74-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm\" (UID: \"ff7f4bf0-e541-4893-a000-027b14e42b74\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"
Oct 30 21:11:54.094738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.094641    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13a507e-48ca-4b4a-90dd-72f30316c75d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7\" (UID: \"a13a507e-48ca-4b4a-90dd-72f30316c75d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"
Oct 30 21:11:54.111700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.111637    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnbtg\" (UniqueName: \"kubernetes.io/projected/a13a507e-48ca-4b4a-90dd-72f30316c75d-kube-api-access-hnbtg\") pod \"payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7\" (UID: \"a13a507e-48ca-4b4a-90dd-72f30316c75d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"
Oct 30 21:11:54.111700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.111668    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ndjj\" (UniqueName: \"kubernetes.io/projected/ff7f4bf0-e541-4893-a000-027b14e42b74-kube-api-access-5ndjj\") pod \"payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm\" (UID: \"ff7f4bf0-e541-4893-a000-027b14e42b74\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"
Oct 30 21:11:54.178789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.178759    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"
Oct 30 21:11:54.219872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.219618    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"
Oct 30 21:11:54.308278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.308249    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:54.398719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.398646    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-socket\") pod \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") "
Oct 30 21:11:54.398878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.398722    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-kube-api-access-btpc2\" (UniqueName: \"kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-kube-api-access-btpc2\") pod \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") "
Oct 30 21:11:54.398878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.398760    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-envoy\") pod \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") "
Oct 30 21:11:54.398878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.398783    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-credential-socket\") pod \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") "
Oct 30 21:11:54.398878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.398819    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-certs\") pod \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") "
Oct 30 21:11:54.398878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.398840    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-podinfo\") pod \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") "
Oct 30 21:11:54.398878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.398873    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-token\" (UniqueName: \"kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-token\") pod \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") "
Oct 30 21:11:54.399184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.398916    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-socket" pod "7d48bcb0-6344-4b05-a92f-272ca78a9fdf" (UID: "7d48bcb0-6344-4b05-a92f-272ca78a9fdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:54.399184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.398926    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-data\") pod \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") "
Oct 30 21:11:54.399184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.399003    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istiod-ca-cert\") pod \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\" (UID: \"7d48bcb0-6344-4b05-a92f-272ca78a9fdf\") "
Oct 30 21:11:54.399184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.399089    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-credential-socket" pod "7d48bcb0-6344-4b05-a92f-272ca78a9fdf" (UID: "7d48bcb0-6344-4b05-a92f-272ca78a9fdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:54.399184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.399174    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-certs" pod "7d48bcb0-6344-4b05-a92f-272ca78a9fdf" (UID: "7d48bcb0-6344-4b05-a92f-272ca78a9fdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:54.399455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.399186    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-data" pod "7d48bcb0-6344-4b05-a92f-272ca78a9fdf" (UID: "7d48bcb0-6344-4b05-a92f-272ca78a9fdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:54.399512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.399463    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istiod-ca-cert" pod "7d48bcb0-6344-4b05-a92f-272ca78a9fdf" (UID: "7d48bcb0-6344-4b05-a92f-272ca78a9fdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:11:54.399512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.399486    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:54.399512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.399506    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:54.399818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.399521    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:54.399818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.399535    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:54.401123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.401099    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-envoy" pod "7d48bcb0-6344-4b05-a92f-272ca78a9fdf" (UID: "7d48bcb0-6344-4b05-a92f-272ca78a9fdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:11:54.401235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.401144    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-kube-api-access-btpc2" pod "7d48bcb0-6344-4b05-a92f-272ca78a9fdf" (UID: "7d48bcb0-6344-4b05-a92f-272ca78a9fdf"). InnerVolumeSpecName "kube-api-access-btpc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:54.401594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.401579    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-podinfo" pod "7d48bcb0-6344-4b05-a92f-272ca78a9fdf" (UID: "7d48bcb0-6344-4b05-a92f-272ca78a9fdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:11:54.401653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.401607    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-token" pod "7d48bcb0-6344-4b05-a92f-272ca78a9fdf" (UID: "7d48bcb0-6344-4b05-a92f-272ca78a9fdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:11:54.500575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.500537    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btpc2\" (UniqueName: \"kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-kube-api-access-btpc2\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:54.500575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.500573    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:54.500575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.500583    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:54.500781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.500592    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:54.500781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.500600    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d48bcb0-6344-4b05-a92f-272ca78a9fdf-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:11:54.514136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.514111    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"]
Oct 30 21:11:54.517186 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:54.517162    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13a507e_48ca_4b4a_90dd_72f30316c75d.slice/crio-6c81488c0d01a980840c4f4f0c3dc4a9df2b6574123d7572ba9c7702779cac0e WatchSource:0}: Error finding container 6c81488c0d01a980840c4f4f0c3dc4a9df2b6574123d7572ba9c7702779cac0e: Status 404 returned error can't find the container with id 6c81488c0d01a980840c4f4f0c3dc4a9df2b6574123d7572ba9c7702779cac0e
Oct 30 21:11:54.547083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:54.547063    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"]
Oct 30 21:11:54.548539 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:11:54.548514    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7f4bf0_e541_4893_a000_027b14e42b74.slice/crio-b51511454e9522e3b0b26f6b9ccfcdfbdb36d7b4d0aa68da9b6616d77d3b43b7 WatchSource:0}: Error finding container b51511454e9522e3b0b26f6b9ccfcdfbdb36d7b4d0aa68da9b6616d77d3b43b7: Status 404 returned error can't find the container with id b51511454e9522e3b0b26f6b9ccfcdfbdb36d7b4d0aa68da9b6616d77d3b43b7
Oct 30 21:11:55.063483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.063451    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7" event={"ID":"a13a507e-48ca-4b4a-90dd-72f30316c75d","Type":"ContainerStarted","Data":"92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342"}
Oct 30 21:11:55.063651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.063508    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7" event={"ID":"a13a507e-48ca-4b4a-90dd-72f30316c75d","Type":"ContainerStarted","Data":"6c81488c0d01a980840c4f4f0c3dc4a9df2b6574123d7572ba9c7702779cac0e"}
Oct 30 21:11:55.063651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.063638    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"
Oct 30 21:11:55.065219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.065196    4918 generic.go:356] "Generic (PLEG): container finished" podID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerID="8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30" exitCode=0
Oct 30 21:11:55.065328 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.065243    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"
Oct 30 21:11:55.065328 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.065263    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" event={"ID":"7d48bcb0-6344-4b05-a92f-272ca78a9fdf","Type":"ContainerDied","Data":"8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30"}
Oct 30 21:11:55.065328 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.065306    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr" event={"ID":"7d48bcb0-6344-4b05-a92f-272ca78a9fdf","Type":"ContainerDied","Data":"cabea3df06a88b5be405b21d62757bcc4e851d6854f408ecebb84afe9ef2aad4"}
Oct 30 21:11:55.065495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.065338    4918 scope.go:122] "RemoveContainer" containerID="8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30"
Oct 30 21:11:55.066857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.066834    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm" event={"ID":"ff7f4bf0-e541-4893-a000-027b14e42b74","Type":"ContainerStarted","Data":"bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c"}
Oct 30 21:11:55.066968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.066875    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm" event={"ID":"ff7f4bf0-e541-4893-a000-027b14e42b74","Type":"ContainerStarted","Data":"b51511454e9522e3b0b26f6b9ccfcdfbdb36d7b4d0aa68da9b6616d77d3b43b7"}
Oct 30 21:11:55.067525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.067053    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"
Oct 30 21:11:55.075804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.075786    4918 scope.go:122] "RemoveContainer" containerID="d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f"
Oct 30 21:11:55.083990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.083976    4918 scope.go:122] "RemoveContainer" containerID="8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30"
Oct 30 21:11:55.084252 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:11:55.084236    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30\": container with ID starting with 8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30 not found: ID does not exist" containerID="8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30"
Oct 30 21:11:55.084290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.084260    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30"} err="failed to get container status \"8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30\": rpc error: code = NotFound desc = could not find container \"8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30\": container with ID starting with 8bf3403c383b32eb2f0502a779caa1d446bd6e378b9da5cb60ed5d03108a0c30 not found: ID does not exist"
Oct 30 21:11:55.084290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.084275    4918 scope.go:122] "RemoveContainer" containerID="d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f"
Oct 30 21:11:55.084530 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:11:55.084514    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f\": container with ID starting with d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f not found: ID does not exist" containerID="d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f"
Oct 30 21:11:55.084576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.084537    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f"} err="failed to get container status \"d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f\": rpc error: code = NotFound desc = could not find container \"d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f\": container with ID starting with d58fbf11423ddcfa44c1dbb2e6befe08e27e9d989564a0401d80209d806c5f1f not found: ID does not exist"
Oct 30 21:11:55.087400 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.087352    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7" podStartSLOduration=2.087341544 podStartE2EDuration="2.087341544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:55.08575511 +0000 UTC m=+628.132314275" watchObservedRunningTime="2026-10-30 21:11:55.087341544 +0000 UTC m=+628.133900689"
Oct 30 21:11:55.104904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.104879    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"]
Oct 30 21:11:55.113263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.113245    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5ee9aa6f-openshift-default-7b5cccff97-s9rlr"]
Oct 30 21:11:55.129746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:55.129704    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm" podStartSLOduration=2.129692249 podStartE2EDuration="2.129692249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:55.128463171 +0000 UTC m=+628.175022319" watchObservedRunningTime="2026-10-30 21:11:55.129692249 +0000 UTC m=+628.176251431"
Oct 30 21:11:57.076598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:11:57.076565    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" path="/var/lib/kubelet/pods/7d48bcb0-6344-4b05-a92f-272ca78a9fdf/volumes"
Oct 30 21:11:59.169161 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:11:59.169116    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:12:00.040951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:00.040915    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"
Oct 30 21:12:00.042254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:00.042234    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"
Oct 30 21:12:03.048580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:03.048540    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:12:03.049077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:03.048605    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:12:04.059633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.059603    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"
Oct 30 21:12:04.169569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.169538    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"]
Oct 30 21:12:04.170218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.170196    4918 cpu_manager.go:414] "containerMap: removing container" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerName="istio-proxy"
Oct 30 21:12:04.170316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.170220    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerName="istio-proxy"
Oct 30 21:12:04.170316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.170241    4918 cpu_manager.go:414] "containerMap: removing container" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerName="istio-proxy"
Oct 30 21:12:04.170316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.170249    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerName="istio-proxy"
Oct 30 21:12:04.170450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.170357    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerName="istio-proxy"
Oct 30 21:12:04.170450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.170371    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d48bcb0-6344-4b05-a92f-272ca78a9fdf" containerName="istio-proxy"
Oct 30 21:12:04.175323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.175298    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.185614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.185322    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-05a342\"/\"default-dockercfg-45nrp\""
Oct 30 21:12:04.207427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.202342    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-05a342\"/\"simulator-w2-05a342-kserve-self-signed-certs\""
Oct 30 21:12:04.222600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.222513    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-05a342\"/\"kube-root-ca.crt\""
Oct 30 21:12:04.222831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.222789    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-05a342\"/\"openshift-service-ca.crt\""
Oct 30 21:12:04.223638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.223451    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"]
Oct 30 21:12:04.242131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.242102    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c3279db-e5d3-4bce-bf19-490161f761a2-tls-certs\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.242255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.242159    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-tmp-dir\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.242297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.242247    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-home\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.242297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.242280    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbnrq\" (UniqueName: \"kubernetes.io/projected/8c3279db-e5d3-4bce-bf19-490161f761a2-kube-api-access-mbnrq\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.242373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.242358    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-model-cache\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.242462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.242445    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-dshm\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.343006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.342918    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-dshm\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.343006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.342967    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c3279db-e5d3-4bce-bf19-490161f761a2-tls-certs\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.343229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.343109    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-tmp-dir\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.343229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.343157    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-home\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.343229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.343187    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbnrq\" (UniqueName: \"kubernetes.io/projected/8c3279db-e5d3-4bce-bf19-490161f761a2-kube-api-access-mbnrq\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.343425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.343248    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-model-cache\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.343560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.343540    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-tmp-dir\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.344057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.344034    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-model-cache\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.344113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.344047    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-home\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.345490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.345467    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-dshm\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.345825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.345805    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c3279db-e5d3-4bce-bf19-490161f761a2-tls-certs\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.353170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.353144    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbnrq\" (UniqueName: \"kubernetes.io/projected/8c3279db-e5d3-4bce-bf19-490161f761a2-kube-api-access-mbnrq\") pod \"simulator-w2-05a342-kserve-5864486c54-tspkg\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") " pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.515902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.515864    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:04.808784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:04.808749    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"]
Oct 30 21:12:05.207695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:05.185553    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" event={"ID":"8c3279db-e5d3-4bce-bf19-490161f761a2","Type":"ContainerStarted","Data":"5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed"}
Oct 30 21:12:05.207695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:05.185604    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" event={"ID":"8c3279db-e5d3-4bce-bf19-490161f761a2","Type":"ContainerStarted","Data":"d5e30a29b94ee846cf6808f1d53a3c781ad545b80883b7cb3bf4c66ac7bcad1c"}
Oct 30 21:12:05.231266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:05.231219    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podStartSLOduration=1.231205503 podStartE2EDuration="1.231205503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:05.228915722 +0000 UTC m=+638.275474888" watchObservedRunningTime="2026-10-30 21:12:05.231205503 +0000 UTC m=+638.277764648"
Oct 30 21:12:05.516415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:05.516348    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:06.076428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:06.076407    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"
Oct 30 21:12:06.076637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:06.076471    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"
Oct 30 21:12:06.193587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:06.193511    4918 generic.go:356] "Generic (PLEG): container finished" podID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerID="6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b" exitCode=0
Oct 30 21:12:06.193748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:06.193597    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" event={"ID":"f23c4448-ef53-4fe3-9b8d-6e69f5711f53","Type":"ContainerDied","Data":"6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b"}
Oct 30 21:12:06.194098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:06.194079    4918 scope.go:122] "RemoveContainer" containerID="6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b"
Oct 30 21:12:06.195420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:06.195370    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:12:06.195558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:06.195454    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:12:06.195558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:06.195486    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:12:07.075579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:07.075537    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:12:07.075996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:07.075600    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:12:07.220035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:07.220003    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" event={"ID":"f23c4448-ef53-4fe3-9b8d-6e69f5711f53","Type":"ContainerStarted","Data":"4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c"}
Oct 30 21:12:07.347977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:07.347893    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:12:07.348171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:07.348149    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:12:07.348233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:07.348193    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 21:12:08.348367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:08.348327    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:12:08.348892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:08.348382    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 21:12:09.348997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:09.348974    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:12:09.349457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:09.349023    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 21:12:10.153206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.153169    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"]
Oct 30 21:12:10.156884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.156859    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.165859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.165827    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bfc50e\"/\"openshift-service-ca.crt\""
Oct 30 21:12:10.166068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.166027    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bfc50e\"/\"kube-root-ca.crt\""
Oct 30 21:12:10.166186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.166110    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bfc50e\"/\"simulator-w3-bfc50e-kserve-self-signed-certs\""
Oct 30 21:12:10.166465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.166450    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bfc50e\"/\"default-dockercfg-5qtcc\""
Oct 30 21:12:10.166551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.166532    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"]
Oct 30 21:12:10.236367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.236337    4918 generic.go:356] "Generic (PLEG): container finished" podID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerID="54b994aa348f3ddd9bc2ba22b8ec7752ed568f0245513d43d7f990f49fd89d5a" exitCode=0
Oct 30 21:12:10.236560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.236416    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" event={"ID":"0c84dfa1-93db-4289-85a3-0182e98b41cf","Type":"ContainerDied","Data":"54b994aa348f3ddd9bc2ba22b8ec7752ed568f0245513d43d7f990f49fd89d5a"}
Oct 30 21:12:10.236832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.236814    4918 scope.go:122] "RemoveContainer" containerID="54b994aa348f3ddd9bc2ba22b8ec7752ed568f0245513d43d7f990f49fd89d5a"
Oct 30 21:12:10.238283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.238256    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:12:10.238368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.238334    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:12:10.238433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.238368    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:12:10.239038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.239021    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-tmp-dir\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.239128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.239063    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-model-cache\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.239128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.239082    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04b824b-1326-4aa0-885a-8997d849f347-tls-certs\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.239275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.239256    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-home\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.239339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.239315    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-dshm\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.239585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.239569    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r776b\" (UniqueName: \"kubernetes.io/projected/d04b824b-1326-4aa0-885a-8997d849f347-kube-api-access-r776b\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.341081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.341055    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-tmp-dir\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.341219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.341102    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-model-cache\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.342131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.341431    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04b824b-1326-4aa0-885a-8997d849f347-tls-certs\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.342131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.341447    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-model-cache\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.342131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.341504    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-tmp-dir\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.342131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.341557    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-home\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.342131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.341617    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-dshm\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.342131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.341661    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r776b\" (UniqueName: \"kubernetes.io/projected/d04b824b-1326-4aa0-885a-8997d849f347-kube-api-access-r776b\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.342131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.342094    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-home\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.345179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.345154    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04b824b-1326-4aa0-885a-8997d849f347-tls-certs\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.345475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.345454    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-dshm\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.348340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.348244    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:12:10.348340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.348302    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 21:12:10.350336 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.350295    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r776b\" (UniqueName: \"kubernetes.io/projected/d04b824b-1326-4aa0-885a-8997d849f347-kube-api-access-r776b\") pod \"simulator-w3-bfc50e-kserve-84df57b68c-fpr89\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") " pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.471782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.471751    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:10.780099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:10.779974    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"]
Oct 30 21:12:10.783509 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:12:10.783475    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd04b824b_1326_4aa0_885a_8997d849f347.slice/crio-f52cc62c98fcfdd185b139442207154ffae88bf7254da81827081d07d250f359 WatchSource:0}: Error finding container f52cc62c98fcfdd185b139442207154ffae88bf7254da81827081d07d250f359: Status 404 returned error can't find the container with id f52cc62c98fcfdd185b139442207154ffae88bf7254da81827081d07d250f359
Oct 30 21:12:11.245615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:11.245571    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" event={"ID":"d04b824b-1326-4aa0-885a-8997d849f347","Type":"ContainerStarted","Data":"1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145"}
Oct 30 21:12:11.245825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:11.245626    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" event={"ID":"d04b824b-1326-4aa0-885a-8997d849f347","Type":"ContainerStarted","Data":"f52cc62c98fcfdd185b139442207154ffae88bf7254da81827081d07d250f359"}
Oct 30 21:12:11.247898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:11.247876    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" event={"ID":"0c84dfa1-93db-4289-85a3-0182e98b41cf","Type":"ContainerStarted","Data":"fbf3d94c7bc7c81cbbfb617130f1fe77048d45bcacea4903644364e7258fd613"}
Oct 30 21:12:11.263327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:11.263289    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podStartSLOduration=1.263274067 podStartE2EDuration="1.263274067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:11.262708846 +0000 UTC m=+644.309267993" watchObservedRunningTime="2026-10-30 21:12:11.263274067 +0000 UTC m=+644.309833213"
Oct 30 21:12:11.348953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:11.348908    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:12:11.349131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:11.348976    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 21:12:11.442413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:11.441565    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:12:11.442413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:11.442147    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:12:11.442413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:11.442195    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:12:11.472473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:11.472442    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:12.438293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:12.438262    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:12:12.438507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:12.438331    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:12:12.458920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:12.458889    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:12:12.459423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:12.459378    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:12:12.461843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:12.461808    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:12:13.316076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.316043    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"]
Oct 30 21:12:13.319849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.319833    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.332026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.332006    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-384164\"/\"kube-root-ca.crt\""
Oct 30 21:12:13.332157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.332142    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-384164\"/\"default-dockercfg-mldjn\""
Oct 30 21:12:13.332209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.332165    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-384164\"/\"simulator-w0-384164-kserve-self-signed-certs\""
Oct 30 21:12:13.338548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.338528    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-384164\"/\"openshift-service-ca.crt\""
Oct 30 21:12:13.342604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.342583    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"]
Oct 30 21:12:13.438522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.438488    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:12:13.438702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.438542    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:12:13.479291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.479266    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-home\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.479769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.479302    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tmp-dir\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.479769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.479480    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-model-cache\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.479769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.479539    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhz2s\" (UniqueName: \"kubernetes.io/projected/d7fe75d1-2b2e-4aaa-b2b5-752796087620-kube-api-access-dhz2s\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.479769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.479599    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-dshm\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.479769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.479662    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tls-certs\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.580478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.580369    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-model-cache\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.580478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.580448    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhz2s\" (UniqueName: \"kubernetes.io/projected/d7fe75d1-2b2e-4aaa-b2b5-752796087620-kube-api-access-dhz2s\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.580926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.580543    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-dshm\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.580926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.580597    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tls-certs\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.580926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.580653    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-home\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.580926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.580691    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tmp-dir\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.581101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.580972    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-model-cache\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.581101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.581057    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-home\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.581193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.581156    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tmp-dir\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.583353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.583328    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-dshm\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.583505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.583489    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tls-certs\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.602572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.602533    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhz2s\" (UniqueName: \"kubernetes.io/projected/d7fe75d1-2b2e-4aaa-b2b5-752796087620-kube-api-access-dhz2s\") pod \"simulator-w0-384164-kserve-856dc7ddf-wndh9\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") " pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.631998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.631965    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:13.930211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:13.930186    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"]
Oct 30 21:12:13.932981 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:12:13.932955    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7fe75d1_2b2e_4aaa_b2b5_752796087620.slice/crio-26ed78cf7c24c50042abe7a863f14d01fae9d5118ad6ffc8fd54ed4dba5d9a3d WatchSource:0}: Error finding container 26ed78cf7c24c50042abe7a863f14d01fae9d5118ad6ffc8fd54ed4dba5d9a3d: Status 404 returned error can't find the container with id 26ed78cf7c24c50042abe7a863f14d01fae9d5118ad6ffc8fd54ed4dba5d9a3d
Oct 30 21:12:14.272846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:14.272811    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" event={"ID":"d7fe75d1-2b2e-4aaa-b2b5-752796087620","Type":"ContainerStarted","Data":"6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1"}
Oct 30 21:12:14.273037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:14.272866    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" event={"ID":"d7fe75d1-2b2e-4aaa-b2b5-752796087620","Type":"ContainerStarted","Data":"26ed78cf7c24c50042abe7a863f14d01fae9d5118ad6ffc8fd54ed4dba5d9a3d"}
Oct 30 21:12:14.312342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:14.312267    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podStartSLOduration=1.312246266 podStartE2EDuration="1.312246266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:14.292880012 +0000 UTC m=+647.339439158" watchObservedRunningTime="2026-10-30 21:12:14.312246266 +0000 UTC m=+647.358805414"
Oct 30 21:12:14.439149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:14.439113    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:12:14.439447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:14.439420    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:12:14.516875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:14.516842    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:14.530090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:14.530014    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:14.632624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:14.632596    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:15.324161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:15.324135    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:12:15.437908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:15.437881    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:12:15.438068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:15.437930    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:12:16.438616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:16.438579    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:12:16.439118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:16.438645    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 30 21:12:17.442492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:17.442457    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:12:17.442913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:17.442722    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:12:17.443438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:17.443424    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:12:20.472650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:20.472621    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:20.486053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:20.486029    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:21.330944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:21.330918    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:12:23.632455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:23.632420    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:23.645640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:23.645616    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:24.345342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.345312    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:12:24.401872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.401828    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"]
Oct 30 21:12:24.406794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.406771    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.418165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.418140    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"]
Oct 30 21:12:24.418351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.418335    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-05a342\"/\"premium-w2-05a342-kserve-self-signed-certs\""
Oct 30 21:12:24.517199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.517169    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-dshm\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.517417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.517228    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfp6r\" (UniqueName: \"kubernetes.io/projected/2ebb1491-7850-4c93-8087-6d325f8f32c2-kube-api-access-kfp6r\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.517417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.517258    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-model-cache\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.517525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.517424    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-tmp-dir\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.517525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.517516    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebb1491-7850-4c93-8087-6d325f8f32c2-tls-certs\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.517593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.517539    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-home\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.618598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.618513    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-dshm\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.618598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.618567    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfp6r\" (UniqueName: \"kubernetes.io/projected/2ebb1491-7850-4c93-8087-6d325f8f32c2-kube-api-access-kfp6r\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.618598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.618589    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-model-cache\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.618890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.618633    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-tmp-dir\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.618890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.618773    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebb1491-7850-4c93-8087-6d325f8f32c2-tls-certs\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.618890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.618814    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-home\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.619019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.618998    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-model-cache\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.619061 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.619043    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-tmp-dir\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.619192 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.619174    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-home\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.620967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.620946    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-dshm\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.621335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.621320    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebb1491-7850-4c93-8087-6d325f8f32c2-tls-certs\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.627208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.627185    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfp6r\" (UniqueName: \"kubernetes.io/projected/2ebb1491-7850-4c93-8087-6d325f8f32c2-kube-api-access-kfp6r\") pod \"premium-w2-05a342-kserve-9d8b876f5-766vw\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") " pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.721995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.721964    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:24.997327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:24.997305    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"]
Oct 30 21:12:24.999354 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:12:24.999325    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ebb1491_7850_4c93_8087_6d325f8f32c2.slice/crio-fac5c9d3c289d7d563cb49e9162053dbe78cac9a701134d49afba00eae2412c1 WatchSource:0}: Error finding container fac5c9d3c289d7d563cb49e9162053dbe78cac9a701134d49afba00eae2412c1: Status 404 returned error can't find the container with id fac5c9d3c289d7d563cb49e9162053dbe78cac9a701134d49afba00eae2412c1
Oct 30 21:12:25.341776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:25.341665    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" event={"ID":"2ebb1491-7850-4c93-8087-6d325f8f32c2","Type":"ContainerStarted","Data":"cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed"}
Oct 30 21:12:25.341776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:25.341732    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" event={"ID":"2ebb1491-7850-4c93-8087-6d325f8f32c2","Type":"ContainerStarted","Data":"fac5c9d3c289d7d563cb49e9162053dbe78cac9a701134d49afba00eae2412c1"}
Oct 30 21:12:25.373458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:25.372720    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podStartSLOduration=1.372698744 podStartE2EDuration="1.372698744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:25.363226509 +0000 UTC m=+658.409785681" watchObservedRunningTime="2026-10-30 21:12:25.372698744 +0000 UTC m=+658.419257889"
Oct 30 21:12:25.722342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:25.722317    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:29.160336 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:12:29.160290    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:12:34.639623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.639586    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"]
Oct 30 21:12:34.645803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.645777    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.652655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.652629    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"]
Oct 30 21:12:34.656678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.656646    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3011c6\"/\"openshift-service-ca.crt\""
Oct 30 21:12:34.656956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.656808    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3011c6\"/\"premium-w4-3011c6-kserve-self-signed-certs\""
Oct 30 21:12:34.656956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.656857    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3011c6\"/\"default-dockercfg-wh42t\""
Oct 30 21:12:34.656956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.656926    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3011c6\"/\"kube-root-ca.crt\""
Oct 30 21:12:34.714379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.714348    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-dshm\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.714569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.714440    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-home\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.714569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.714505    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-tmp-dir\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.714569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.714550    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxl9\" (UniqueName: \"kubernetes.io/projected/1417521f-e732-4114-b645-b51b1c731cd6-kube-api-access-zrxl9\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.714696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.714649    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1417521f-e732-4114-b645-b51b1c731cd6-tls-certs\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.714737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.714714    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-model-cache\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.722591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.722569    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:34.734993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.734974    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:34.817006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.816964    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-dshm\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.817271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.817235    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-home\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.817454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.817306    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-tmp-dir\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.817454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.817358    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxl9\" (UniqueName: \"kubernetes.io/projected/1417521f-e732-4114-b645-b51b1c731cd6-kube-api-access-zrxl9\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.817578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.817485    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1417521f-e732-4114-b645-b51b1c731cd6-tls-certs\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.817637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.817577    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-model-cache\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.818084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.818050    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-model-cache\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.818658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.818634    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-tmp-dir\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.818875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.818854    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-home\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.821354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.821327    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-dshm\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.821677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.821655    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1417521f-e732-4114-b645-b51b1c731cd6-tls-certs\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.827892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.827872    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxl9\" (UniqueName: \"kubernetes.io/projected/1417521f-e732-4114-b645-b51b1c731cd6-kube-api-access-zrxl9\") pod \"premium-w4-3011c6-kserve-d4985fdbb-cb2tj\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") " pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:34.961683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:34.961592    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:35.237557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:35.237532    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"]
Oct 30 21:12:35.239474 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:12:35.239446    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1417521f_e732_4114_b645_b51b1c731cd6.slice/crio-51c492a4d36dacb218f95405b1113bc0c8b90404f481e19cba353bb8c023953b WatchSource:0}: Error finding container 51c492a4d36dacb218f95405b1113bc0c8b90404f481e19cba353bb8c023953b: Status 404 returned error can't find the container with id 51c492a4d36dacb218f95405b1113bc0c8b90404f481e19cba353bb8c023953b
Oct 30 21:12:35.394407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:35.394359    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" event={"ID":"1417521f-e732-4114-b645-b51b1c731cd6","Type":"ContainerStarted","Data":"d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587"}
Oct 30 21:12:35.394580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:35.394420    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" event={"ID":"1417521f-e732-4114-b645-b51b1c731cd6","Type":"ContainerStarted","Data":"51c492a4d36dacb218f95405b1113bc0c8b90404f481e19cba353bb8c023953b"}
Oct 30 21:12:35.405695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:35.405668    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:12:35.414973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:35.414934    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podStartSLOduration=1.414921443 podStartE2EDuration="1.414921443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:35.412368005 +0000 UTC m=+668.458927162" watchObservedRunningTime="2026-10-30 21:12:35.414921443 +0000 UTC m=+668.461480589"
Oct 30 21:12:35.962148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:35.962114    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:44.341398 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.341301    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"]
Oct 30 21:12:44.346064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.346045    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.365904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.365881    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a33b12\"/\"premium-w1-a33b12-kserve-self-signed-certs\""
Oct 30 21:12:44.366178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.366137    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"]
Oct 30 21:12:44.367426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.367410    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a33b12\"/\"default-dockercfg-p666d\""
Oct 30 21:12:44.367519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.367432    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a33b12\"/\"openshift-service-ca.crt\""
Oct 30 21:12:44.367519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.367448    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a33b12\"/\"kube-root-ca.crt\""
Oct 30 21:12:44.440837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.440809    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-dshm\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.440837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.440837    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-model-cache\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.441051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.440948    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-home\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.441051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.440999    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tls-certs\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.441168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.441107    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46mkr\" (UniqueName: \"kubernetes.io/projected/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-kube-api-access-46mkr\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.441224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.441194    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tmp-dir\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.542793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.542756    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-home\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.542793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.542801    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tls-certs\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.543039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.542837    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46mkr\" (UniqueName: \"kubernetes.io/projected/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-kube-api-access-46mkr\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.543039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.542892    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tmp-dir\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.543039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.542930    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-dshm\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.543039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.542948    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-model-cache\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.543259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.543218    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-home\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.543324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.543306    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tmp-dir\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.543477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.543420    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-model-cache\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.545251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.545221    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-dshm\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.545531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.545515    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tls-certs\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.551770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.551750    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46mkr\" (UniqueName: \"kubernetes.io/projected/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-kube-api-access-46mkr\") pod \"premium-w1-a33b12-kserve-6db59766f9-72fx4\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") " pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.658877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.658803    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:44.937292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.937260    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"]
Oct 30 21:12:44.938610 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:12:44.938582    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2e6ce17_cac4_42ab_9a05_d233ea2d9392.slice/crio-923c4ee4b5f1a0d82606ebd7c4d4d68aaa490195ce79efec07b0a67a2bd612f9 WatchSource:0}: Error finding container 923c4ee4b5f1a0d82606ebd7c4d4d68aaa490195ce79efec07b0a67a2bd612f9: Status 404 returned error can't find the container with id 923c4ee4b5f1a0d82606ebd7c4d4d68aaa490195ce79efec07b0a67a2bd612f9
Oct 30 21:12:44.961815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.961796    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:44.974763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:44.974743    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:45.461137 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:45.461103    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" event={"ID":"f2e6ce17-cac4-42ab-9a05-d233ea2d9392","Type":"ContainerStarted","Data":"ad7a1d0aee6c5115ba59cf87f53146415cbbee5ef314f60fca0a084e82c43226"}
Oct 30 21:12:45.461579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:45.461146    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" event={"ID":"f2e6ce17-cac4-42ab-9a05-d233ea2d9392","Type":"ContainerStarted","Data":"923c4ee4b5f1a0d82606ebd7c4d4d68aaa490195ce79efec07b0a67a2bd612f9"}
Oct 30 21:12:45.472450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:45.472431    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:12:45.481101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:45.481060    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podStartSLOduration=1.481047967 podStartE2EDuration="1.481047967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:45.479459571 +0000 UTC m=+678.526018729" watchObservedRunningTime="2026-10-30 21:12:45.481047967 +0000 UTC m=+678.527607109"
Oct 30 21:12:45.659800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:45.659765    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:53.093643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.093612    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"]
Oct 30 21:12:53.097726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.097703    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.108026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.107995    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-05a342-openshift-default-dockercfg-vxlf4\""
Oct 30 21:12:53.110637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.110618    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"]
Oct 30 21:12:53.247754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.247727    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.247950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.247763    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.247950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.247841    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.247950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.247929    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv86p\" (UniqueName: \"kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-kube-api-access-fv86p\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.248118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.248056    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.248118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.248089    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.248216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.248159    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.248216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.248180    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.248311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.248240    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.349596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.349504    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.349831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.349812    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.349979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.349964    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.350098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.350085    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv86p\" (UniqueName: \"kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-kube-api-access-fv86p\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.350278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.350263    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.350416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.350378    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.350544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.350530    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.350656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.350643    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.350753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.350741    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.352089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.352062    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.352499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.352460    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.352620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.352571    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.353330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.353303    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.353478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.353335    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.355023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.355000    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.355240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.355223    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.361345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.361319    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.362513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.362492    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv86p\" (UniqueName: \"kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-kube-api-access-fv86p\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.413907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.413076    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:53.710627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.710598    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"]
Oct 30 21:12:53.712185 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:12:53.712153    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a6cc5b_be20_4ad9_ab5b_e60852ee21b5.slice/crio-8a92c5bf5d77f426581e4f524c33af1b638471b6f297a86002c34799a4fc7723 WatchSource:0}: Error finding container 8a92c5bf5d77f426581e4f524c33af1b638471b6f297a86002c34799a4fc7723: Status 404 returned error can't find the container with id 8a92c5bf5d77f426581e4f524c33af1b638471b6f297a86002c34799a4fc7723
Oct 30 21:12:53.714646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.714612    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:12:53.714759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.714687    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:12:53.714759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:53.714730    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:12:54.538943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:54.538909    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2" event={"ID":"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5","Type":"ContainerStarted","Data":"29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b"}
Oct 30 21:12:54.539309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:54.538966    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2" event={"ID":"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5","Type":"ContainerStarted","Data":"8a92c5bf5d77f426581e4f524c33af1b638471b6f297a86002c34799a4fc7723"}
Oct 30 21:12:54.567495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:54.567451    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2" podStartSLOduration=1.5674386949999999 podStartE2EDuration="1.567438695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:54.565043038 +0000 UTC m=+687.611602192" watchObservedRunningTime="2026-10-30 21:12:54.567438695 +0000 UTC m=+687.613997858"
Oct 30 21:12:54.659785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:54.659748    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:54.672744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:54.672708    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:55.413477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:55.413428    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:12:55.416430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:55.415542    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:12:55.416430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:55.415599    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2" podUID="c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 21:12:55.556131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:55.556101    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:12:56.414198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:56.414158    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:12:56.414405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:56.414216    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2" podUID="c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 21:12:57.413984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.413951    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:12:57.414469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.413997    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2" podUID="c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 21:12:57.650941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.650907    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"]
Oct 30 21:12:57.654839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.654816    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.665458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.665401    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bfc50e-openshift-default-dockercfg-g92dm\""
Oct 30 21:12:57.671339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.671316    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"]
Oct 30 21:12:57.720662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.720612    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.720863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.720724    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.720863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.720751    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.720863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.720771    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.720863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.720793    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.720863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.720843    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02880084-8170-487e-9fdf-0e9a3c7ba287-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.721150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.720941    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94dg5\" (UniqueName: \"kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-kube-api-access-94dg5\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.721150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.721014    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.721150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.721058    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.822671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.822632    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.822671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.822675    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.822905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.822711    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.822905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.822740    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.822905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.822756    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.822905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.822771    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.822905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.822803    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.822905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.822836    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02880084-8170-487e-9fdf-0e9a3c7ba287-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.822905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.822899    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94dg5\" (UniqueName: \"kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-kube-api-access-94dg5\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.823293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.823139    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.823293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.823196    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.823358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.823313    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.823481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.823465    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.823998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.823976    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02880084-8170-487e-9fdf-0e9a3c7ba287-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.825377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.825348    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.825507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.825487    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.832617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.832597    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.832724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.832662    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94dg5\" (UniqueName: \"kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-kube-api-access-94dg5\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:57.969082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:57.968999    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:58.281290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:58.281246    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"]
Oct 30 21:12:58.283703 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:12:58.283671    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02880084_8170_487e_9fdf_0e9a3c7ba287.slice/crio-b49d44d664ab2dbedae4fdd42bd1649104e7eabe969888f90e8effe2b569e7c4 WatchSource:0}: Error finding container b49d44d664ab2dbedae4fdd42bd1649104e7eabe969888f90e8effe2b569e7c4: Status 404 returned error can't find the container with id b49d44d664ab2dbedae4fdd42bd1649104e7eabe969888f90e8effe2b569e7c4
Oct 30 21:12:58.306981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:58.302219    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:12:58.306981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:58.302317    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:12:58.306981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:58.302370    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:12:58.422413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:58.417806    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:12:58.422413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:58.417876    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2" podUID="c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 21:12:58.564463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:58.563642    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" event={"ID":"02880084-8170-487e-9fdf-0e9a3c7ba287","Type":"ContainerStarted","Data":"b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee"}
Oct 30 21:12:58.564463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:58.563705    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" event={"ID":"02880084-8170-487e-9fdf-0e9a3c7ba287","Type":"ContainerStarted","Data":"b49d44d664ab2dbedae4fdd42bd1649104e7eabe969888f90e8effe2b569e7c4"}
Oct 30 21:12:58.609413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:58.605822    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" podStartSLOduration=1.605801884 podStartE2EDuration="1.605801884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:58.605697252 +0000 UTC m=+691.652256401" watchObservedRunningTime="2026-10-30 21:12:58.605801884 +0000 UTC m=+691.652361030"
Oct 30 21:12:58.971570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:58.971543    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:12:59.184526 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:12:59.184492    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:12:59.414474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:59.414013    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:12:59.414474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:59.414117    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2" podUID="c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 21:12:59.970761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:59.970727    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:12:59.971175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:12:59.970802    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" podUID="02880084-8170-487e-9fdf-0e9a3c7ba287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 21:13:00.476752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:00.476724    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:13:00.477115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:00.477092    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:13:00.477974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:00.477955    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:13:00.969795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:00.969710    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:13:00.969795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:00.969776    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" podUID="02880084-8170-487e-9fdf-0e9a3c7ba287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 21:13:01.969886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:01.969851    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:13:01.970277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:01.969922    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" podUID="02880084-8170-487e-9fdf-0e9a3c7ba287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 21:13:02.972839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:02.972805    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:13:02.973467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:02.973436    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" podUID="02880084-8170-487e-9fdf-0e9a3c7ba287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 21:13:03.969948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:03.969913    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:13:03.970140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:03.969969    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" podUID="02880084-8170-487e-9fdf-0e9a3c7ba287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 21:13:05.219159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:05.219127    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:13:05.220138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:05.220120    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:13:05.221045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:05.221022    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:13:06.561566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.561534    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"]
Oct 30 21:13:06.566054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.566038    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:06.572885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.572864    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:06.573027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.572873    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:13:06.573137 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.572882    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:13:06.573241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.572920    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:13:06.574438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.574416    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"]
Oct 30 21:13:06.673097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.673066    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d76c6f06-z7b4z\" (UID: \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:06.673275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.673202    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-785ll\" (UniqueName: \"kubernetes.io/projected/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-kube-api-access-785ll\") pod \"maas-api-revoke-sub-d76c6f06-z7b4z\" (UID: \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:06.673275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.673258    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d76c6f06-z7b4z\" (UID: \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:06.774792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.774764    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d76c6f06-z7b4z\" (UID: \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:06.774964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.774825    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-785ll\" (UniqueName: \"kubernetes.io/projected/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-kube-api-access-785ll\") pod \"maas-api-revoke-sub-d76c6f06-z7b4z\" (UID: \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:06.774964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.774864    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d76c6f06-z7b4z\" (UID: \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:06.775535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.775511    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d76c6f06-z7b4z\" (UID: \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:06.777276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.777258    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d76c6f06-z7b4z\" (UID: \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:06.784181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.784157    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-785ll\" (UniqueName: \"kubernetes.io/projected/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-kube-api-access-785ll\") pod \"maas-api-revoke-sub-d76c6f06-z7b4z\" (UID: \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:06.890466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.886635    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:06.907179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.907152    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"]
Oct 30 21:13:06.911099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.911078    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:06.919033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.919010    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"]
Oct 30 21:13:06.977422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.976581    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm58j\" (UniqueName: \"kubernetes.io/projected/4c5a8708-252d-4451-839f-49802615aa93-kube-api-access-cm58j\") pod \"maas-api-revoke-sub-eb2fa007-dphkr\" (UID: \"4c5a8708-252d-4451-839f-49802615aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:06.977422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.976667    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5a8708-252d-4451-839f-49802615aa93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb2fa007-dphkr\" (UID: \"4c5a8708-252d-4451-839f-49802615aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:06.977422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:06.976748    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5a8708-252d-4451-839f-49802615aa93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb2fa007-dphkr\" (UID: \"4c5a8708-252d-4451-839f-49802615aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:07.078073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.078047    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5a8708-252d-4451-839f-49802615aa93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb2fa007-dphkr\" (UID: \"4c5a8708-252d-4451-839f-49802615aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:07.078236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.078122    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm58j\" (UniqueName: \"kubernetes.io/projected/4c5a8708-252d-4451-839f-49802615aa93-kube-api-access-cm58j\") pod \"maas-api-revoke-sub-eb2fa007-dphkr\" (UID: \"4c5a8708-252d-4451-839f-49802615aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:07.078236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.078209    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5a8708-252d-4451-839f-49802615aa93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb2fa007-dphkr\" (UID: \"4c5a8708-252d-4451-839f-49802615aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:07.078874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.078849    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5a8708-252d-4451-839f-49802615aa93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb2fa007-dphkr\" (UID: \"4c5a8708-252d-4451-839f-49802615aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:07.080659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.080628    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5a8708-252d-4451-839f-49802615aa93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb2fa007-dphkr\" (UID: \"4c5a8708-252d-4451-839f-49802615aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:07.087620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.087586    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm58j\" (UniqueName: \"kubernetes.io/projected/4c5a8708-252d-4451-839f-49802615aa93-kube-api-access-cm58j\") pod \"maas-api-revoke-sub-eb2fa007-dphkr\" (UID: \"4c5a8708-252d-4451-839f-49802615aa93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:07.176607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.176577    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"]
Oct 30 21:13:07.178426 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:13:07.178384    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cf7a0e_4830_4d88_9bf7_a1774d8a662b.slice/crio-e1a5e2334e2deb015836c3f163bd70c4dd6a3c216e1094344ca449eaa9cbb7d4 WatchSource:0}: Error finding container e1a5e2334e2deb015836c3f163bd70c4dd6a3c216e1094344ca449eaa9cbb7d4: Status 404 returned error can't find the container with id e1a5e2334e2deb015836c3f163bd70c4dd6a3c216e1094344ca449eaa9cbb7d4
Oct 30 21:13:07.245140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.245113    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:07.541668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.541642    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"]
Oct 30 21:13:07.543046 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:13:07.543021    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5a8708_252d_4451_839f_49802615aa93.slice/crio-c0cc3af4c670884932faf3750fe826a27d98c5cb0ee354ba42c6d952160cd5cc WatchSource:0}: Error finding container c0cc3af4c670884932faf3750fe826a27d98c5cb0ee354ba42c6d952160cd5cc: Status 404 returned error can't find the container with id c0cc3af4c670884932faf3750fe826a27d98c5cb0ee354ba42c6d952160cd5cc
Oct 30 21:13:07.639185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.639150    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z" event={"ID":"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b","Type":"ContainerStarted","Data":"046f26493012766ca4d959ca46bb88adde0035e0f92c06bd5633d6c9167f4896"}
Oct 30 21:13:07.639627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.639209    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z" event={"ID":"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b","Type":"ContainerStarted","Data":"e1a5e2334e2deb015836c3f163bd70c4dd6a3c216e1094344ca449eaa9cbb7d4"}
Oct 30 21:13:07.640450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.640419    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr" event={"ID":"4c5a8708-252d-4451-839f-49802615aa93","Type":"ContainerStarted","Data":"c0cc3af4c670884932faf3750fe826a27d98c5cb0ee354ba42c6d952160cd5cc"}
Oct 30 21:13:07.656990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:07.656950    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z" podStartSLOduration=1.656936816 podStartE2EDuration="1.656936816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:07.656491837 +0000 UTC m=+700.703050999" watchObservedRunningTime="2026-10-30 21:13:07.656936816 +0000 UTC m=+700.703495960"
Oct 30 21:13:08.648315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:08.648230    4918 generic.go:356] "Generic (PLEG): container finished" podID="b8cf7a0e-4830-4d88-9bf7-a1774d8a662b" containerID="046f26493012766ca4d959ca46bb88adde0035e0f92c06bd5633d6c9167f4896" exitCode=0
Oct 30 21:13:08.648714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:08.648314    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z" event={"ID":"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b","Type":"ContainerDied","Data":"046f26493012766ca4d959ca46bb88adde0035e0f92c06bd5633d6c9167f4896"}
Oct 30 21:13:08.649864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:08.649835    4918 generic.go:356] "Generic (PLEG): container finished" podID="4c5a8708-252d-4451-839f-49802615aa93" containerID="fd1ec70b0336da6f8da79fb2b6fea11e9ad70c2ea6cc3d82796674a3fe27214e" exitCode=0
Oct 30 21:13:08.649981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:08.649911    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr" event={"ID":"4c5a8708-252d-4451-839f-49802615aa93","Type":"ContainerDied","Data":"fd1ec70b0336da6f8da79fb2b6fea11e9ad70c2ea6cc3d82796674a3fe27214e"}
Oct 30 21:13:09.854261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.854239    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:09.857640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.857622    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:09.929124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.929041    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-xks-ca\") pod \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\" (UID: \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\") "
Oct 30 21:13:09.929124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.929116    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-openshift-ca\") pod \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\" (UID: \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\") "
Oct 30 21:13:09.929298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.929249    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5a8708-252d-4451-839f-49802615aa93-kube-api-access-cm58j\" (UniqueName: \"kubernetes.io/projected/4c5a8708-252d-4451-839f-49802615aa93-kube-api-access-cm58j\") pod \"4c5a8708-252d-4451-839f-49802615aa93\" (UID: \"4c5a8708-252d-4451-839f-49802615aa93\") "
Oct 30 21:13:09.929298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.929278    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c5a8708-252d-4451-839f-49802615aa93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5a8708-252d-4451-839f-49802615aa93-maas-api-xks-ca\") pod \"4c5a8708-252d-4451-839f-49802615aa93\" (UID: \"4c5a8708-252d-4451-839f-49802615aa93\") "
Oct 30 21:13:09.929363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.929311    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-kube-api-access-785ll\" (UniqueName: \"kubernetes.io/projected/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-kube-api-access-785ll\") pod \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\" (UID: \"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b\") "
Oct 30 21:13:09.929484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.929468    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5a8708-252d-4451-839f-49802615aa93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5a8708-252d-4451-839f-49802615aa93-maas-api-openshift-ca\") pod \"4c5a8708-252d-4451-839f-49802615aa93\" (UID: \"4c5a8708-252d-4451-839f-49802615aa93\") "
Oct 30 21:13:09.930641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.930601    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-openshift-ca" pod "b8cf7a0e-4830-4d88-9bf7-a1774d8a662b" (UID: "b8cf7a0e-4830-4d88-9bf7-a1774d8a662b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:09.930773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.930702    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5a8708-252d-4451-839f-49802615aa93-maas-api-openshift-ca" pod "4c5a8708-252d-4451-839f-49802615aa93" (UID: "4c5a8708-252d-4451-839f-49802615aa93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:09.933298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.933257    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-kube-api-access-785ll" pod "b8cf7a0e-4830-4d88-9bf7-a1774d8a662b" (UID: "b8cf7a0e-4830-4d88-9bf7-a1774d8a662b"). InnerVolumeSpecName "kube-api-access-785ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:09.933298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.933281    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-xks-ca" pod "b8cf7a0e-4830-4d88-9bf7-a1774d8a662b" (UID: "b8cf7a0e-4830-4d88-9bf7-a1774d8a662b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:09.933298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.933287    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5a8708-252d-4451-839f-49802615aa93-kube-api-access-cm58j" pod "4c5a8708-252d-4451-839f-49802615aa93" (UID: "4c5a8708-252d-4451-839f-49802615aa93"). InnerVolumeSpecName "kube-api-access-cm58j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:09.933554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:09.933513    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5a8708-252d-4451-839f-49802615aa93-maas-api-xks-ca" pod "4c5a8708-252d-4451-839f-49802615aa93" (UID: "4c5a8708-252d-4451-839f-49802615aa93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:10.040416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:10.031510    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5a8708-252d-4451-839f-49802615aa93-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.040416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:10.031545    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.040416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:10.031560    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.040416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:10.031576    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm58j\" (UniqueName: \"kubernetes.io/projected/4c5a8708-252d-4451-839f-49802615aa93-kube-api-access-cm58j\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.040416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:10.031590    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5a8708-252d-4451-839f-49802615aa93-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.040416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:10.031607    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-785ll\" (UniqueName: \"kubernetes.io/projected/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b-kube-api-access-785ll\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:10.661974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:10.661939    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z" event={"ID":"b8cf7a0e-4830-4d88-9bf7-a1774d8a662b","Type":"ContainerDied","Data":"e1a5e2334e2deb015836c3f163bd70c4dd6a3c216e1094344ca449eaa9cbb7d4"}
Oct 30 21:13:10.661974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:10.661965    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"
Oct 30 21:13:10.662241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:10.662024    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1a5e2334e2deb015836c3f163bd70c4dd6a3c216e1094344ca449eaa9cbb7d4"
Oct 30 21:13:10.663342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:10.663317    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr" event={"ID":"4c5a8708-252d-4451-839f-49802615aa93","Type":"ContainerDied","Data":"c0cc3af4c670884932faf3750fe826a27d98c5cb0ee354ba42c6d952160cd5cc"}
Oct 30 21:13:10.663477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:10.663356    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0cc3af4c670884932faf3750fe826a27d98c5cb0ee354ba42c6d952160cd5cc"
Oct 30 21:13:10.663477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:10.663326    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"
Oct 30 21:13:14.612995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.612953    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"]
Oct 30 21:13:14.613435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.613420    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4c5a8708-252d-4451-839f-49802615aa93" containerName="revoke-keys"
Oct 30 21:13:14.613493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.613436    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5a8708-252d-4451-839f-49802615aa93" containerName="revoke-keys"
Oct 30 21:13:14.613493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.613456    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b8cf7a0e-4830-4d88-9bf7-a1774d8a662b" containerName="revoke-keys"
Oct 30 21:13:14.613493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.613464    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cf7a0e-4830-4d88-9bf7-a1774d8a662b" containerName="revoke-keys"
Oct 30 21:13:14.613589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.613564    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5a8708-252d-4451-839f-49802615aa93" containerName="revoke-keys"
Oct 30 21:13:14.613589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.613573    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cf7a0e-4830-4d88-9bf7-a1774d8a662b" containerName="revoke-keys"
Oct 30 21:13:14.617009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.616990    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.621188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.621168    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3011c6-openshift-default-dockercfg-m96xs\""
Oct 30 21:13:14.629133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.629112    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"]
Oct 30 21:13:14.713666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.713634    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.713854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.713695    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.713854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.713789    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.713854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.713839    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.714027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.713915    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.714027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.713960    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.714027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.713982    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvslr\" (UniqueName: \"kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-kube-api-access-rvslr\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.714188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.714130    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.714188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.714166    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bed962-7c0d-4881-b889-bd4d5402a6de-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.816796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.816753    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.816989 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.816850    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.816989 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.816895    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.816989 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.816953    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.817150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.817000    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.817150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.817049    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.817150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.817084    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvslr\" (UniqueName: \"kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-kube-api-access-rvslr\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.817279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.817181    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.817279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.817205    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bed962-7c0d-4881-b889-bd4d5402a6de-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.818109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.818077    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bed962-7c0d-4881-b889-bd4d5402a6de-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.818752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.818726    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.823226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.823202    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.823371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.823313    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.823371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.823333    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.824162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.824141    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.825193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.825171    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.827690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.827667    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.834457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.834435    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvslr\" (UniqueName: \"kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-kube-api-access-rvslr\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:14.932694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:14.932609    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:15.211598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.211571    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"]
Oct 30 21:13:15.213038 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:13:15.213009    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28bed962_7c0d_4881_b889_bd4d5402a6de.slice/crio-f773f9dcdedd4193a3cf7911a24a8b25a0a5892fbcd23d84d337455467823f8d WatchSource:0}: Error finding container f773f9dcdedd4193a3cf7911a24a8b25a0a5892fbcd23d84d337455467823f8d: Status 404 returned error can't find the container with id f773f9dcdedd4193a3cf7911a24a8b25a0a5892fbcd23d84d337455467823f8d
Oct 30 21:13:15.215201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.215170    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:13:15.215272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.215237    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:13:15.215307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.215270    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:13:15.276897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.276866    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"]
Oct 30 21:13:15.283003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.282973    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:15.291127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.290202    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:13:15.291127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.290271    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:15.291127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.290216    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:13:15.291127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.290909    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:13:15.296832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.296814    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"]
Oct 30 21:13:15.426939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.426896    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7486dadd-x5qr5\" (UID: \"af55126d-0189-44ab-892b-4b363d3738cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:15.427135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.427116    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7486dadd-x5qr5\" (UID: \"af55126d-0189-44ab-892b-4b363d3738cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:15.427190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.427151    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlp7g\" (UniqueName: \"kubernetes.io/projected/af55126d-0189-44ab-892b-4b363d3738cd-kube-api-access-jlp7g\") pod \"maas-api-revoke-sub-7486dadd-x5qr5\" (UID: \"af55126d-0189-44ab-892b-4b363d3738cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:15.528016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.527982    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7486dadd-x5qr5\" (UID: \"af55126d-0189-44ab-892b-4b363d3738cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:15.528016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.528022    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlp7g\" (UniqueName: \"kubernetes.io/projected/af55126d-0189-44ab-892b-4b363d3738cd-kube-api-access-jlp7g\") pod \"maas-api-revoke-sub-7486dadd-x5qr5\" (UID: \"af55126d-0189-44ab-892b-4b363d3738cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:15.528277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.528068    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7486dadd-x5qr5\" (UID: \"af55126d-0189-44ab-892b-4b363d3738cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:15.528832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.528807    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7486dadd-x5qr5\" (UID: \"af55126d-0189-44ab-892b-4b363d3738cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:15.530570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.530549    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7486dadd-x5qr5\" (UID: \"af55126d-0189-44ab-892b-4b363d3738cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:15.536987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.536962    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlp7g\" (UniqueName: \"kubernetes.io/projected/af55126d-0189-44ab-892b-4b363d3738cd-kube-api-access-jlp7g\") pod \"maas-api-revoke-sub-7486dadd-x5qr5\" (UID: \"af55126d-0189-44ab-892b-4b363d3738cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:15.609229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.609198    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:15.697273 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.697226    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5" event={"ID":"28bed962-7c0d-4881-b889-bd4d5402a6de","Type":"ContainerStarted","Data":"97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17"}
Oct 30 21:13:15.697752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.697306    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5" event={"ID":"28bed962-7c0d-4881-b889-bd4d5402a6de","Type":"ContainerStarted","Data":"f773f9dcdedd4193a3cf7911a24a8b25a0a5892fbcd23d84d337455467823f8d"}
Oct 30 21:13:15.746553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.746492    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5" podStartSLOduration=1.746476855 podStartE2EDuration="1.746476855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:15.744127115 +0000 UTC m=+708.790686286" watchObservedRunningTime="2026-10-30 21:13:15.746476855 +0000 UTC m=+708.793036000"
Oct 30 21:13:15.910920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.910650    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"]
Oct 30 21:13:15.915641 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:13:15.915605    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf55126d_0189_44ab_892b_4b363d3738cd.slice/crio-ffec99e3c881d5d128a46181519e95c75442b2dde4f93d425e659ffc0c4ab4f4 WatchSource:0}: Error finding container ffec99e3c881d5d128a46181519e95c75442b2dde4f93d425e659ffc0c4ab4f4: Status 404 returned error can't find the container with id ffec99e3c881d5d128a46181519e95c75442b2dde4f93d425e659ffc0c4ab4f4
Oct 30 21:13:15.933693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:15.933021    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:16.716798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:16.716709    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5" event={"ID":"af55126d-0189-44ab-892b-4b363d3738cd","Type":"ContainerStarted","Data":"82ee1a63a770912fe4cedadc9d48ccc29f7a053d76d0a5a9f42dc1a723ff127c"}
Oct 30 21:13:16.716798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:16.716761    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5" event={"ID":"af55126d-0189-44ab-892b-4b363d3738cd","Type":"ContainerStarted","Data":"ffec99e3c881d5d128a46181519e95c75442b2dde4f93d425e659ffc0c4ab4f4"}
Oct 30 21:13:16.734133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:16.734086    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5" podStartSLOduration=1.7340746870000001 podStartE2EDuration="1.734074687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:16.732409291 +0000 UTC m=+709.778968446" watchObservedRunningTime="2026-10-30 21:13:16.734074687 +0000 UTC m=+709.780633831"
Oct 30 21:13:16.934408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:16.934359    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:13:16.934547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:16.934438    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5" podUID="28bed962-7c0d-4881-b889-bd4d5402a6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 21:13:17.737514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:17.737478    4918 generic.go:356] "Generic (PLEG): container finished" podID="af55126d-0189-44ab-892b-4b363d3738cd" containerID="82ee1a63a770912fe4cedadc9d48ccc29f7a053d76d0a5a9f42dc1a723ff127c" exitCode=0
Oct 30 21:13:17.737999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:17.737758    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5" event={"ID":"af55126d-0189-44ab-892b-4b363d3738cd","Type":"ContainerDied","Data":"82ee1a63a770912fe4cedadc9d48ccc29f7a053d76d0a5a9f42dc1a723ff127c"}
Oct 30 21:13:17.933127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:17.933096    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:13:17.933330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:17.933150    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5" podUID="28bed962-7c0d-4881-b889-bd4d5402a6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 21:13:18.933149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:18.933124    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:13:18.933491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:18.933174    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5" podUID="28bed962-7c0d-4881-b889-bd4d5402a6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 21:13:18.938662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:18.938643    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:18.972232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:18.972205    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-openshift-ca\") pod \"af55126d-0189-44ab-892b-4b363d3738cd\" (UID: \"af55126d-0189-44ab-892b-4b363d3738cd\") "
Oct 30 21:13:18.972368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:18.972237    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-xks-ca\") pod \"af55126d-0189-44ab-892b-4b363d3738cd\" (UID: \"af55126d-0189-44ab-892b-4b363d3738cd\") "
Oct 30 21:13:18.972368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:18.972309    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af55126d-0189-44ab-892b-4b363d3738cd-kube-api-access-jlp7g\" (UniqueName: \"kubernetes.io/projected/af55126d-0189-44ab-892b-4b363d3738cd-kube-api-access-jlp7g\") pod \"af55126d-0189-44ab-892b-4b363d3738cd\" (UID: \"af55126d-0189-44ab-892b-4b363d3738cd\") "
Oct 30 21:13:18.972651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:18.972618    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-openshift-ca" pod "af55126d-0189-44ab-892b-4b363d3738cd" (UID: "af55126d-0189-44ab-892b-4b363d3738cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:18.972867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:18.972850    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:18.975117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:18.975090    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-xks-ca" pod "af55126d-0189-44ab-892b-4b363d3738cd" (UID: "af55126d-0189-44ab-892b-4b363d3738cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:18.975200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:18.975116    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af55126d-0189-44ab-892b-4b363d3738cd-kube-api-access-jlp7g" pod "af55126d-0189-44ab-892b-4b363d3738cd" (UID: "af55126d-0189-44ab-892b-4b363d3738cd"). InnerVolumeSpecName "kube-api-access-jlp7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:19.073695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.073663    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af55126d-0189-44ab-892b-4b363d3738cd-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:19.073915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.073895    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlp7g\" (UniqueName: \"kubernetes.io/projected/af55126d-0189-44ab-892b-4b363d3738cd-kube-api-access-jlp7g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:19.749860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.749823    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5" event={"ID":"af55126d-0189-44ab-892b-4b363d3738cd","Type":"ContainerDied","Data":"ffec99e3c881d5d128a46181519e95c75442b2dde4f93d425e659ffc0c4ab4f4"}
Oct 30 21:13:19.750019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.749883    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffec99e3c881d5d128a46181519e95c75442b2dde4f93d425e659ffc0c4ab4f4"
Oct 30 21:13:19.750019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.749938    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"
Oct 30 21:13:19.832580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.832546    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"]
Oct 30 21:13:19.833034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.833020    4918 cpu_manager.go:414] "containerMap: removing container" podUID="af55126d-0189-44ab-892b-4b363d3738cd" containerName="revoke-keys"
Oct 30 21:13:19.833083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.833039    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af55126d-0189-44ab-892b-4b363d3738cd" containerName="revoke-keys"
Oct 30 21:13:19.833155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.833146    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="af55126d-0189-44ab-892b-4b363d3738cd" containerName="revoke-keys"
Oct 30 21:13:19.836481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.836467    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:19.841485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.841435    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:19.844012 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.843993    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:13:19.844579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.844557    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:13:19.844699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.844612    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:13:19.846278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.846259    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"]
Oct 30 21:13:19.884640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.884597    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sd87\" (UniqueName: \"kubernetes.io/projected/93a11411-a922-4fd1-82c6-0f34f3dd2500-kube-api-access-8sd87\") pod \"maas-api-revoke-sub-191380e4-fvbj2\" (UID: \"93a11411-a922-4fd1-82c6-0f34f3dd2500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:19.884830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.884711    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-191380e4-fvbj2\" (UID: \"93a11411-a922-4fd1-82c6-0f34f3dd2500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:19.884830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.884793    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-191380e4-fvbj2\" (UID: \"93a11411-a922-4fd1-82c6-0f34f3dd2500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:19.940823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.940779    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:13:19.941479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.941445    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5" podUID="28bed962-7c0d-4881-b889-bd4d5402a6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 21:13:19.990129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.990088    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sd87\" (UniqueName: \"kubernetes.io/projected/93a11411-a922-4fd1-82c6-0f34f3dd2500-kube-api-access-8sd87\") pod \"maas-api-revoke-sub-191380e4-fvbj2\" (UID: \"93a11411-a922-4fd1-82c6-0f34f3dd2500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:19.990786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:19.990360    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"]
Oct 30 21:13:20.035101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.035028    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-191380e4-fvbj2\" (UID: \"93a11411-a922-4fd1-82c6-0f34f3dd2500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:20.035481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.035449    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-191380e4-fvbj2\" (UID: \"93a11411-a922-4fd1-82c6-0f34f3dd2500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:20.062712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.061840    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-191380e4-fvbj2\" (UID: \"93a11411-a922-4fd1-82c6-0f34f3dd2500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:20.070173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.070127    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-191380e4-fvbj2\" (UID: \"93a11411-a922-4fd1-82c6-0f34f3dd2500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:20.079427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.078236    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sd87\" (UniqueName: \"kubernetes.io/projected/93a11411-a922-4fd1-82c6-0f34f3dd2500-kube-api-access-8sd87\") pod \"maas-api-revoke-sub-191380e4-fvbj2\" (UID: \"93a11411-a922-4fd1-82c6-0f34f3dd2500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:20.080003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.079976    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"]
Oct 30 21:13:20.080501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.080464    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:20.136564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.136533    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38a5f909-c96w7\" (UID: \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:20.136690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.136585    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38a5f909-c96w7\" (UID: \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:20.136690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.136634    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgs5k\" (UniqueName: \"kubernetes.io/projected/ce2f1586-4ce8-4898-a3be-06b6fd562a33-kube-api-access-kgs5k\") pod \"maas-api-revoke-sub-38a5f909-c96w7\" (UID: \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:20.152490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.152465    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:20.247415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.246079    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38a5f909-c96w7\" (UID: \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:20.247415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.246151    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38a5f909-c96w7\" (UID: \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:20.247415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.246208    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgs5k\" (UniqueName: \"kubernetes.io/projected/ce2f1586-4ce8-4898-a3be-06b6fd562a33-kube-api-access-kgs5k\") pod \"maas-api-revoke-sub-38a5f909-c96w7\" (UID: \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:20.247711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.247568    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38a5f909-c96w7\" (UID: \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:20.280096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.280066    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgs5k\" (UniqueName: \"kubernetes.io/projected/ce2f1586-4ce8-4898-a3be-06b6fd562a33-kube-api-access-kgs5k\") pod \"maas-api-revoke-sub-38a5f909-c96w7\" (UID: \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:20.283931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.281752    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38a5f909-c96w7\" (UID: \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:20.451418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.450620    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:20.645842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.645760    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"]
Oct 30 21:13:20.879469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.879371    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2" event={"ID":"93a11411-a922-4fd1-82c6-0f34f3dd2500","Type":"ContainerStarted","Data":"77609de1faf550a08499ffbbf01d7e7d750164128f646476c68cc23b018c02d7"}
Oct 30 21:13:20.933572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.933052    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:13:20.933572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:20.933107    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5" podUID="28bed962-7c0d-4881-b889-bd4d5402a6de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 30 21:13:21.117737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:21.117708    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"]
Oct 30 21:13:21.888471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:21.888364    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7" event={"ID":"ce2f1586-4ce8-4898-a3be-06b6fd562a33","Type":"ContainerStarted","Data":"e669515a782e0d89c65a3b9549beb8cfb39886fcf9bdf6cd6c5cd9f7f3ad571d"}
Oct 30 21:13:21.888471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:21.888441    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7" event={"ID":"ce2f1586-4ce8-4898-a3be-06b6fd562a33","Type":"ContainerStarted","Data":"9a552ee08115ca9a802d0f19e0e72d0647bd4bc38ba56f8eabbc5fe63f6e41b9"}
Oct 30 21:13:21.890848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:21.890813    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2" event={"ID":"93a11411-a922-4fd1-82c6-0f34f3dd2500","Type":"ContainerStarted","Data":"8868a2ae6f98ce1316d9a76c99d405b658d8134905ea2df49d161cd40688d3ba"}
Oct 30 21:13:21.907995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:21.907947    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7" podStartSLOduration=2.9079304759999998 podStartE2EDuration="2.907930476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:21.90485304 +0000 UTC m=+714.951412203" watchObservedRunningTime="2026-10-30 21:13:21.907930476 +0000 UTC m=+714.954489623"
Oct 30 21:13:21.970887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:21.970845    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:21.971266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:21.971233    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:21.972836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:21.972816    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:13:21.992748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:21.992699    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2" podStartSLOduration=2.99268292 podStartE2EDuration="2.99268292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:21.921584629 +0000 UTC m=+714.968143786" watchObservedRunningTime="2026-10-30 21:13:21.99268292 +0000 UTC m=+715.039242064"
Oct 30 21:13:22.897399 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:22.897367    4918 generic.go:356] "Generic (PLEG): container finished" podID="93a11411-a922-4fd1-82c6-0f34f3dd2500" containerID="8868a2ae6f98ce1316d9a76c99d405b658d8134905ea2df49d161cd40688d3ba" exitCode=0
Oct 30 21:13:22.897813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:22.897454    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2" event={"ID":"93a11411-a922-4fd1-82c6-0f34f3dd2500","Type":"ContainerDied","Data":"8868a2ae6f98ce1316d9a76c99d405b658d8134905ea2df49d161cd40688d3ba"}
Oct 30 21:13:22.899027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:22.899003    4918 generic.go:356] "Generic (PLEG): container finished" podID="ce2f1586-4ce8-4898-a3be-06b6fd562a33" containerID="e669515a782e0d89c65a3b9549beb8cfb39886fcf9bdf6cd6c5cd9f7f3ad571d" exitCode=0
Oct 30 21:13:22.899130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:22.899072    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7" event={"ID":"ce2f1586-4ce8-4898-a3be-06b6fd562a33","Type":"ContainerDied","Data":"e669515a782e0d89c65a3b9549beb8cfb39886fcf9bdf6cd6c5cd9f7f3ad571d"}
Oct 30 21:13:24.058891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.058855    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:24.093325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.093299    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:24.162044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.161970    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a11411-a922-4fd1-82c6-0f34f3dd2500-kube-api-access-8sd87\" (UniqueName: \"kubernetes.io/projected/93a11411-a922-4fd1-82c6-0f34f3dd2500-kube-api-access-8sd87\") pod \"93a11411-a922-4fd1-82c6-0f34f3dd2500\" (UID: \"93a11411-a922-4fd1-82c6-0f34f3dd2500\") "
Oct 30 21:13:24.162044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.162041    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-openshift-ca\") pod \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\" (UID: \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\") "
Oct 30 21:13:24.162232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.162085    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-xks-ca\") pod \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\" (UID: \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\") "
Oct 30 21:13:24.162232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.162121    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-openshift-ca\") pod \"93a11411-a922-4fd1-82c6-0f34f3dd2500\" (UID: \"93a11411-a922-4fd1-82c6-0f34f3dd2500\") "
Oct 30 21:13:24.162232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.162178    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2f1586-4ce8-4898-a3be-06b6fd562a33-kube-api-access-kgs5k\" (UniqueName: \"kubernetes.io/projected/ce2f1586-4ce8-4898-a3be-06b6fd562a33-kube-api-access-kgs5k\") pod \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\" (UID: \"ce2f1586-4ce8-4898-a3be-06b6fd562a33\") "
Oct 30 21:13:24.162366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.162277    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-xks-ca\") pod \"93a11411-a922-4fd1-82c6-0f34f3dd2500\" (UID: \"93a11411-a922-4fd1-82c6-0f34f3dd2500\") "
Oct 30 21:13:24.162533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.162492    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-openshift-ca" pod "ce2f1586-4ce8-4898-a3be-06b6fd562a33" (UID: "ce2f1586-4ce8-4898-a3be-06b6fd562a33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:24.162533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.162511    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-openshift-ca" pod "93a11411-a922-4fd1-82c6-0f34f3dd2500" (UID: "93a11411-a922-4fd1-82c6-0f34f3dd2500"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:24.162825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.162790    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:24.162825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.162816    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:24.164465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.164440    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a11411-a922-4fd1-82c6-0f34f3dd2500-kube-api-access-8sd87" pod "93a11411-a922-4fd1-82c6-0f34f3dd2500" (UID: "93a11411-a922-4fd1-82c6-0f34f3dd2500"). InnerVolumeSpecName "kube-api-access-8sd87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:24.164581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.164496    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-xks-ca" pod "ce2f1586-4ce8-4898-a3be-06b6fd562a33" (UID: "ce2f1586-4ce8-4898-a3be-06b6fd562a33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:24.164713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.164691    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-xks-ca" pod "93a11411-a922-4fd1-82c6-0f34f3dd2500" (UID: "93a11411-a922-4fd1-82c6-0f34f3dd2500"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:24.164824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.164709    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2f1586-4ce8-4898-a3be-06b6fd562a33-kube-api-access-kgs5k" pod "ce2f1586-4ce8-4898-a3be-06b6fd562a33" (UID: "ce2f1586-4ce8-4898-a3be-06b6fd562a33"). InnerVolumeSpecName "kube-api-access-kgs5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:24.263379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.263342    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgs5k\" (UniqueName: \"kubernetes.io/projected/ce2f1586-4ce8-4898-a3be-06b6fd562a33-kube-api-access-kgs5k\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:24.263379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.263370    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93a11411-a922-4fd1-82c6-0f34f3dd2500-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:24.263379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.263380    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sd87\" (UniqueName: \"kubernetes.io/projected/93a11411-a922-4fd1-82c6-0f34f3dd2500-kube-api-access-8sd87\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:24.263379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.263404    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce2f1586-4ce8-4898-a3be-06b6fd562a33-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:24.924441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.922727    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"
Oct 30 21:13:24.924441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.922740    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7" event={"ID":"ce2f1586-4ce8-4898-a3be-06b6fd562a33","Type":"ContainerDied","Data":"9a552ee08115ca9a802d0f19e0e72d0647bd4bc38ba56f8eabbc5fe63f6e41b9"}
Oct 30 21:13:24.924441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.922846    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a552ee08115ca9a802d0f19e0e72d0647bd4bc38ba56f8eabbc5fe63f6e41b9"
Oct 30 21:13:24.934312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.930692    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"
Oct 30 21:13:24.934312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.930736    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2" event={"ID":"93a11411-a922-4fd1-82c6-0f34f3dd2500","Type":"ContainerDied","Data":"77609de1faf550a08499ffbbf01d7e7d750164128f646476c68cc23b018c02d7"}
Oct 30 21:13:24.934312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:24.930786    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77609de1faf550a08499ffbbf01d7e7d750164128f646476c68cc23b018c02d7"
Oct 30 21:13:24.951274 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:13:24.951245    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2f1586_4ce8_4898_a3be_06b6fd562a33.slice/crio-9a552ee08115ca9a802d0f19e0e72d0647bd4bc38ba56f8eabbc5fe63f6e41b9\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a11411_a922_4fd1_82c6_0f34f3dd2500.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2f1586_4ce8_4898_a3be_06b6fd562a33.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:13:24.951413 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:13:24.951250    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2f1586_4ce8_4898_a3be_06b6fd562a33.slice/crio-9a552ee08115ca9a802d0f19e0e72d0647bd4bc38ba56f8eabbc5fe63f6e41b9\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a11411_a922_4fd1_82c6_0f34f3dd2500.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2f1586_4ce8_4898_a3be_06b6fd562a33.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:13:29.181997 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:13:29.181958    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:13:32.376659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.376626    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"]
Oct 30 21:13:32.377377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.377352    4918 cpu_manager.go:414] "containerMap: removing container" podUID="93a11411-a922-4fd1-82c6-0f34f3dd2500" containerName="revoke-keys"
Oct 30 21:13:32.377712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.377692    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a11411-a922-4fd1-82c6-0f34f3dd2500" containerName="revoke-keys"
Oct 30 21:13:32.377852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.377827    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ce2f1586-4ce8-4898-a3be-06b6fd562a33" containerName="revoke-keys"
Oct 30 21:13:32.377962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.377950    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2f1586-4ce8-4898-a3be-06b6fd562a33" containerName="revoke-keys"
Oct 30 21:13:32.378240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.378208    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a11411-a922-4fd1-82c6-0f34f3dd2500" containerName="revoke-keys"
Oct 30 21:13:32.378240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.378239    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2f1586-4ce8-4898-a3be-06b6fd562a33" containerName="revoke-keys"
Oct 30 21:13:32.381870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.381852    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:32.387358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.387248    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:13:32.387358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.387312    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:32.388140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.388117    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"]
Oct 30 21:13:32.388681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.388666    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:13:32.388788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.388691    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:13:32.533965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.533924    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fc074ef-xd6w9\" (UID: \"13997d71-c50a-4b3b-bf63-e52399720783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:32.534131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.534009    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdzx\" (UniqueName: \"kubernetes.io/projected/13997d71-c50a-4b3b-bf63-e52399720783-kube-api-access-zgdzx\") pod \"maas-api-revoke-sub-0fc074ef-xd6w9\" (UID: \"13997d71-c50a-4b3b-bf63-e52399720783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:32.534131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.534084    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fc074ef-xd6w9\" (UID: \"13997d71-c50a-4b3b-bf63-e52399720783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:32.635961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.635880    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fc074ef-xd6w9\" (UID: \"13997d71-c50a-4b3b-bf63-e52399720783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:32.635961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.635919    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdzx\" (UniqueName: \"kubernetes.io/projected/13997d71-c50a-4b3b-bf63-e52399720783-kube-api-access-zgdzx\") pod \"maas-api-revoke-sub-0fc074ef-xd6w9\" (UID: \"13997d71-c50a-4b3b-bf63-e52399720783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:32.636175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.635977    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fc074ef-xd6w9\" (UID: \"13997d71-c50a-4b3b-bf63-e52399720783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:32.636641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.636615    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0fc074ef-xd6w9\" (UID: \"13997d71-c50a-4b3b-bf63-e52399720783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:32.638614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.638597    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0fc074ef-xd6w9\" (UID: \"13997d71-c50a-4b3b-bf63-e52399720783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:32.644605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.644588    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdzx\" (UniqueName: \"kubernetes.io/projected/13997d71-c50a-4b3b-bf63-e52399720783-kube-api-access-zgdzx\") pod \"maas-api-revoke-sub-0fc074ef-xd6w9\" (UID: \"13997d71-c50a-4b3b-bf63-e52399720783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:32.701653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.701625    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:32.976186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:32.976160    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"]
Oct 30 21:13:32.977885 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:13:32.977858    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13997d71_c50a_4b3b_bf63_e52399720783.slice/crio-2c817074c95addb9c72a13423b9441c5049b3256a301f22cf83fb834bdcbe711 WatchSource:0}: Error finding container 2c817074c95addb9c72a13423b9441c5049b3256a301f22cf83fb834bdcbe711: Status 404 returned error can't find the container with id 2c817074c95addb9c72a13423b9441c5049b3256a301f22cf83fb834bdcbe711
Oct 30 21:13:33.985154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:33.985124    4918 generic.go:356] "Generic (PLEG): container finished" podID="13997d71-c50a-4b3b-bf63-e52399720783" containerID="d602fbd04912bf4cef54dfdfd2e19ba89d46a674c13632b836edc3017477c6f5" exitCode=0
Oct 30 21:13:33.985539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:33.985213    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9" event={"ID":"13997d71-c50a-4b3b-bf63-e52399720783","Type":"ContainerDied","Data":"d602fbd04912bf4cef54dfdfd2e19ba89d46a674c13632b836edc3017477c6f5"}
Oct 30 21:13:33.985539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:33.985260    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9" event={"ID":"13997d71-c50a-4b3b-bf63-e52399720783","Type":"ContainerStarted","Data":"2c817074c95addb9c72a13423b9441c5049b3256a301f22cf83fb834bdcbe711"}
Oct 30 21:13:35.148316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.148292    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:35.264865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.264792    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-xks-ca\") pod \"13997d71-c50a-4b3b-bf63-e52399720783\" (UID: \"13997d71-c50a-4b3b-bf63-e52399720783\") "
Oct 30 21:13:35.265021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.264880    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-openshift-ca\") pod \"13997d71-c50a-4b3b-bf63-e52399720783\" (UID: \"13997d71-c50a-4b3b-bf63-e52399720783\") "
Oct 30 21:13:35.265021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.264992    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13997d71-c50a-4b3b-bf63-e52399720783-kube-api-access-zgdzx\" (UniqueName: \"kubernetes.io/projected/13997d71-c50a-4b3b-bf63-e52399720783-kube-api-access-zgdzx\") pod \"13997d71-c50a-4b3b-bf63-e52399720783\" (UID: \"13997d71-c50a-4b3b-bf63-e52399720783\") "
Oct 30 21:13:35.265259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.265234    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-openshift-ca" pod "13997d71-c50a-4b3b-bf63-e52399720783" (UID: "13997d71-c50a-4b3b-bf63-e52399720783"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:35.265448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.265432    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:35.267143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.267119    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-xks-ca" pod "13997d71-c50a-4b3b-bf63-e52399720783" (UID: "13997d71-c50a-4b3b-bf63-e52399720783"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:35.267187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.267171    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13997d71-c50a-4b3b-bf63-e52399720783-kube-api-access-zgdzx" pod "13997d71-c50a-4b3b-bf63-e52399720783" (UID: "13997d71-c50a-4b3b-bf63-e52399720783"). InnerVolumeSpecName "kube-api-access-zgdzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:35.366949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.366921    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgdzx\" (UniqueName: \"kubernetes.io/projected/13997d71-c50a-4b3b-bf63-e52399720783-kube-api-access-zgdzx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:35.366949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.366947    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13997d71-c50a-4b3b-bf63-e52399720783-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:35.788887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.788859    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"]
Oct 30 21:13:35.789342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.789330    4918 cpu_manager.go:414] "containerMap: removing container" podUID="13997d71-c50a-4b3b-bf63-e52399720783" containerName="revoke-keys"
Oct 30 21:13:35.789383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.789344    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13997d71-c50a-4b3b-bf63-e52399720783" containerName="revoke-keys"
Oct 30 21:13:35.789460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.789451    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="13997d71-c50a-4b3b-bf63-e52399720783" containerName="revoke-keys"
Oct 30 21:13:35.793099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.793080    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:35.800139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.800101    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"]
Oct 30 21:13:35.873260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.873232    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b26e13e-pjrwc\" (UID: \"19565d3c-4c31-4118-bcc9-8420563f488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:35.873442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.873269    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b26e13e-pjrwc\" (UID: \"19565d3c-4c31-4118-bcc9-8420563f488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:35.873442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.873375    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9slrp\" (UniqueName: \"kubernetes.io/projected/19565d3c-4c31-4118-bcc9-8420563f488f-kube-api-access-9slrp\") pod \"maas-api-revoke-sub-2b26e13e-pjrwc\" (UID: \"19565d3c-4c31-4118-bcc9-8420563f488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:35.974920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.974886    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b26e13e-pjrwc\" (UID: \"19565d3c-4c31-4118-bcc9-8420563f488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:35.975063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.974927    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b26e13e-pjrwc\" (UID: \"19565d3c-4c31-4118-bcc9-8420563f488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:35.975063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.974966    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9slrp\" (UniqueName: \"kubernetes.io/projected/19565d3c-4c31-4118-bcc9-8420563f488f-kube-api-access-9slrp\") pod \"maas-api-revoke-sub-2b26e13e-pjrwc\" (UID: \"19565d3c-4c31-4118-bcc9-8420563f488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:35.975712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.975691    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b26e13e-pjrwc\" (UID: \"19565d3c-4c31-4118-bcc9-8420563f488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:35.977376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.977356    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b26e13e-pjrwc\" (UID: \"19565d3c-4c31-4118-bcc9-8420563f488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:35.982535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.982513    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9slrp\" (UniqueName: \"kubernetes.io/projected/19565d3c-4c31-4118-bcc9-8420563f488f-kube-api-access-9slrp\") pod \"maas-api-revoke-sub-2b26e13e-pjrwc\" (UID: \"19565d3c-4c31-4118-bcc9-8420563f488f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:35.997848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.997821    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"
Oct 30 21:13:35.997931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.997859    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9" event={"ID":"13997d71-c50a-4b3b-bf63-e52399720783","Type":"ContainerDied","Data":"2c817074c95addb9c72a13423b9441c5049b3256a301f22cf83fb834bdcbe711"}
Oct 30 21:13:35.997931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:35.997900    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c817074c95addb9c72a13423b9441c5049b3256a301f22cf83fb834bdcbe711"
Oct 30 21:13:36.108456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:36.108348    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:36.419741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:36.407925    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"]
Oct 30 21:13:37.032008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:37.031972    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc" event={"ID":"19565d3c-4c31-4118-bcc9-8420563f488f","Type":"ContainerStarted","Data":"f8762aae30d1afcdea023df1329b23808f670be1ebe60e76f4b37e919ec4cafe"}
Oct 30 21:13:37.033165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:37.033139    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc" event={"ID":"19565d3c-4c31-4118-bcc9-8420563f488f","Type":"ContainerStarted","Data":"1402c413e8d382f331424dce6fd81a9b458010cab27b8abf2361077d9699d4b6"}
Oct 30 21:13:37.066998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:37.066890    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc" podStartSLOduration=2.06685749 podStartE2EDuration="2.06685749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:37.064825904 +0000 UTC m=+730.111385047" watchObservedRunningTime="2026-10-30 21:13:37.06685749 +0000 UTC m=+730.113416635"
Oct 30 21:13:38.059413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.058854    4918 generic.go:356] "Generic (PLEG): container finished" podID="19565d3c-4c31-4118-bcc9-8420563f488f" containerID="f8762aae30d1afcdea023df1329b23808f670be1ebe60e76f4b37e919ec4cafe" exitCode=0
Oct 30 21:13:38.059413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.058943    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc" event={"ID":"19565d3c-4c31-4118-bcc9-8420563f488f","Type":"ContainerDied","Data":"f8762aae30d1afcdea023df1329b23808f670be1ebe60e76f4b37e919ec4cafe"}
Oct 30 21:13:38.193442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.192508    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"]
Oct 30 21:13:38.201415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.199709    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.335462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.328186    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.335462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.328264    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.335462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.328317    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6521302f-c44e-40f0-b342-10c81833e27a-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.335462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.328353    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.335462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.328446    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnqv\" (UniqueName: \"kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-kube-api-access-qcnqv\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.335462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.328484    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.335462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.328512    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6521302f-c44e-40f0-b342-10c81833e27a-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.335462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.328589    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.335462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.328661    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.383416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.382455    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"]
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.452494    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.452567    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.452621    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6521302f-c44e-40f0-b342-10c81833e27a-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.452658    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.452726    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnqv\" (UniqueName: \"kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-kube-api-access-qcnqv\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.452762    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.452796    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6521302f-c44e-40f0-b342-10c81833e27a-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.452866    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.452939    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.453483    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.454079    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.454130    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.467682    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.471276    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.472088    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6521302f-c44e-40f0-b342-10c81833e27a-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.472436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.472171    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnqv\" (UniqueName: \"kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-kube-api-access-qcnqv\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.473799 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.473182    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6521302f-c44e-40f0-b342-10c81833e27a-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.475273 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.475249    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:38.634462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:38.633661    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:39.028431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.024473    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"]
Oct 30 21:13:39.111451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:13:39.091076    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6521302f_c44e_40f0_b342_10c81833e27a.slice/crio-a15db8875bd42687ceae9736aa5b90f3f1ada299a8d5ae2562589090a0fd36c7 WatchSource:0}: Error finding container a15db8875bd42687ceae9736aa5b90f3f1ada299a8d5ae2562589090a0fd36c7: Status 404 returned error can't find the container with id a15db8875bd42687ceae9736aa5b90f3f1ada299a8d5ae2562589090a0fd36c7
Oct 30 21:13:39.112103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.111497    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:13:39.112103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.111595    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:13:39.112103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.111644    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:13:39.758574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.758552    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:39.844292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.844264    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-openshift-ca\") pod \"19565d3c-4c31-4118-bcc9-8420563f488f\" (UID: \"19565d3c-4c31-4118-bcc9-8420563f488f\") "
Oct 30 21:13:39.844514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.844346    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-xks-ca\") pod \"19565d3c-4c31-4118-bcc9-8420563f488f\" (UID: \"19565d3c-4c31-4118-bcc9-8420563f488f\") "
Oct 30 21:13:39.844592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.844542    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19565d3c-4c31-4118-bcc9-8420563f488f-kube-api-access-9slrp\" (UniqueName: \"kubernetes.io/projected/19565d3c-4c31-4118-bcc9-8420563f488f-kube-api-access-9slrp\") pod \"19565d3c-4c31-4118-bcc9-8420563f488f\" (UID: \"19565d3c-4c31-4118-bcc9-8420563f488f\") "
Oct 30 21:13:39.844731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.844708    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-openshift-ca" pod "19565d3c-4c31-4118-bcc9-8420563f488f" (UID: "19565d3c-4c31-4118-bcc9-8420563f488f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:39.844888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.844871    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:39.846656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.846632    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-xks-ca" pod "19565d3c-4c31-4118-bcc9-8420563f488f" (UID: "19565d3c-4c31-4118-bcc9-8420563f488f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:39.846768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.846709    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19565d3c-4c31-4118-bcc9-8420563f488f-kube-api-access-9slrp" pod "19565d3c-4c31-4118-bcc9-8420563f488f" (UID: "19565d3c-4c31-4118-bcc9-8420563f488f"). InnerVolumeSpecName "kube-api-access-9slrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:39.945615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.945585    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9slrp\" (UniqueName: \"kubernetes.io/projected/19565d3c-4c31-4118-bcc9-8420563f488f-kube-api-access-9slrp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:39.945615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:39.945614    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19565d3c-4c31-4118-bcc9-8420563f488f-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:40.140188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:40.140113    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"
Oct 30 21:13:40.140636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:40.140112    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc" event={"ID":"19565d3c-4c31-4118-bcc9-8420563f488f","Type":"ContainerDied","Data":"1402c413e8d382f331424dce6fd81a9b458010cab27b8abf2361077d9699d4b6"}
Oct 30 21:13:40.140636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:40.140234    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1402c413e8d382f331424dce6fd81a9b458010cab27b8abf2361077d9699d4b6"
Oct 30 21:13:40.142144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:40.142112    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" event={"ID":"6521302f-c44e-40f0-b342-10c81833e27a","Type":"ContainerStarted","Data":"8c5cc264f3f94f219169be1bb5546b804acfa72aaaa18905f59fc495f308afc2"}
Oct 30 21:13:40.142287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:40.142164    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" event={"ID":"6521302f-c44e-40f0-b342-10c81833e27a","Type":"ContainerStarted","Data":"a15db8875bd42687ceae9736aa5b90f3f1ada299a8d5ae2562589090a0fd36c7"}
Oct 30 21:13:40.166794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:40.166648    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" podStartSLOduration=2.166635675 podStartE2EDuration="2.166635675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:40.166526485 +0000 UTC m=+733.213085630" watchObservedRunningTime="2026-10-30 21:13:40.166635675 +0000 UTC m=+733.213194821"
Oct 30 21:13:40.635690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:40.635649    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:40.637242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:40.637216    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:13:40.637313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:40.637264    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" podUID="6521302f-c44e-40f0-b342-10c81833e27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 21:13:41.635842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:41.635806    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:13:41.636271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:41.635859    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" podUID="6521302f-c44e-40f0-b342-10c81833e27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 21:13:42.636004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:42.635970    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:13:42.636436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:42.636030    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" podUID="6521302f-c44e-40f0-b342-10c81833e27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 21:13:43.636314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:43.636280    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:13:43.636771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:43.636335    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" podUID="6521302f-c44e-40f0-b342-10c81833e27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 21:13:44.636856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:44.636813    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:13:44.637341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:44.636884    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" podUID="6521302f-c44e-40f0-b342-10c81833e27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 21:13:45.636591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:45.636551    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:13:45.636775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:45.636623    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" podUID="6521302f-c44e-40f0-b342-10c81833e27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 21:13:46.134550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.134508    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"]
Oct 30 21:13:46.135187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.135166    4918 cpu_manager.go:414] "containerMap: removing container" podUID="19565d3c-4c31-4118-bcc9-8420563f488f" containerName="revoke-keys"
Oct 30 21:13:46.135281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.135190    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19565d3c-4c31-4118-bcc9-8420563f488f" containerName="revoke-keys"
Oct 30 21:13:46.135337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.135314    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="19565d3c-4c31-4118-bcc9-8420563f488f" containerName="revoke-keys"
Oct 30 21:13:46.138644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.138624    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:46.144245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.144222    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:13:46.144992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.144723    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:13:46.144992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.144849    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:13:46.144992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.144957    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:13:46.148509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.148479    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"]
Oct 30 21:13:46.254537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.254496    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8v92\" (UniqueName: \"kubernetes.io/projected/e201359f-1235-4bf4-a085-b88dc962e9b2-kube-api-access-q8v92\") pod \"maas-api-revoke-sub-83f16a28-t9mbd\" (UID: \"e201359f-1235-4bf4-a085-b88dc962e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:46.254739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.254698    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f16a28-t9mbd\" (UID: \"e201359f-1235-4bf4-a085-b88dc962e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:46.254814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.254796    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f16a28-t9mbd\" (UID: \"e201359f-1235-4bf4-a085-b88dc962e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:46.356406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.356358    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f16a28-t9mbd\" (UID: \"e201359f-1235-4bf4-a085-b88dc962e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:46.356605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.356443    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f16a28-t9mbd\" (UID: \"e201359f-1235-4bf4-a085-b88dc962e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:46.356605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.356488    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8v92\" (UniqueName: \"kubernetes.io/projected/e201359f-1235-4bf4-a085-b88dc962e9b2-kube-api-access-q8v92\") pod \"maas-api-revoke-sub-83f16a28-t9mbd\" (UID: \"e201359f-1235-4bf4-a085-b88dc962e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:46.357237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.357212    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83f16a28-t9mbd\" (UID: \"e201359f-1235-4bf4-a085-b88dc962e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:46.359301 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.359279    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83f16a28-t9mbd\" (UID: \"e201359f-1235-4bf4-a085-b88dc962e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:46.366278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.366252    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8v92\" (UniqueName: \"kubernetes.io/projected/e201359f-1235-4bf4-a085-b88dc962e9b2-kube-api-access-q8v92\") pod \"maas-api-revoke-sub-83f16a28-t9mbd\" (UID: \"e201359f-1235-4bf4-a085-b88dc962e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:46.458473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.457738    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:46.642839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.642542    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:46.645904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.643306    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:46.645904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.645129    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:13:46.881777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:46.880854    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"]
Oct 30 21:13:46.882729 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:13:46.882698    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode201359f_1235_4bf4_a085_b88dc962e9b2.slice/crio-292dcb0c0dc45bc6dd211bcc762c4e6618c15bf3a3546945eab68784b229fa41 WatchSource:0}: Error finding container 292dcb0c0dc45bc6dd211bcc762c4e6618c15bf3a3546945eab68784b229fa41: Status 404 returned error can't find the container with id 292dcb0c0dc45bc6dd211bcc762c4e6618c15bf3a3546945eab68784b229fa41
Oct 30 21:13:47.192048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:47.191963    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd" event={"ID":"e201359f-1235-4bf4-a085-b88dc962e9b2","Type":"ContainerStarted","Data":"292dcb0c0dc45bc6dd211bcc762c4e6618c15bf3a3546945eab68784b229fa41"}
Oct 30 21:13:48.242431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:48.235909    4918 generic.go:356] "Generic (PLEG): container finished" podID="e201359f-1235-4bf4-a085-b88dc962e9b2" containerID="ed8b563d1ee2e41e3656606905c7e1a8721855600f1f868f3b745ce1a1407c87" exitCode=0
Oct 30 21:13:48.250417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:48.244973    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd" event={"ID":"e201359f-1235-4bf4-a085-b88dc962e9b2","Type":"ContainerDied","Data":"ed8b563d1ee2e41e3656606905c7e1a8721855600f1f868f3b745ce1a1407c87"}
Oct 30 21:13:49.592066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:49.592044    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:49.760445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:49.760413    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-openshift-ca\") pod \"e201359f-1235-4bf4-a085-b88dc962e9b2\" (UID: \"e201359f-1235-4bf4-a085-b88dc962e9b2\") "
Oct 30 21:13:49.760613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:49.760584    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-xks-ca\") pod \"e201359f-1235-4bf4-a085-b88dc962e9b2\" (UID: \"e201359f-1235-4bf4-a085-b88dc962e9b2\") "
Oct 30 21:13:49.760669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:49.760620    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e201359f-1235-4bf4-a085-b88dc962e9b2-kube-api-access-q8v92\" (UniqueName: \"kubernetes.io/projected/e201359f-1235-4bf4-a085-b88dc962e9b2-kube-api-access-q8v92\") pod \"e201359f-1235-4bf4-a085-b88dc962e9b2\" (UID: \"e201359f-1235-4bf4-a085-b88dc962e9b2\") "
Oct 30 21:13:49.760669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:49.760657    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-openshift-ca" pod "e201359f-1235-4bf4-a085-b88dc962e9b2" (UID: "e201359f-1235-4bf4-a085-b88dc962e9b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:13:49.761062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:49.761043    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:49.762770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:49.762747    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-xks-ca" pod "e201359f-1235-4bf4-a085-b88dc962e9b2" (UID: "e201359f-1235-4bf4-a085-b88dc962e9b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:13:49.762909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:49.762895    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e201359f-1235-4bf4-a085-b88dc962e9b2-kube-api-access-q8v92" pod "e201359f-1235-4bf4-a085-b88dc962e9b2" (UID: "e201359f-1235-4bf4-a085-b88dc962e9b2"). InnerVolumeSpecName "kube-api-access-q8v92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:13:49.862454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:49.862369    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e201359f-1235-4bf4-a085-b88dc962e9b2-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:49.862454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:49.862415    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8v92\" (UniqueName: \"kubernetes.io/projected/e201359f-1235-4bf4-a085-b88dc962e9b2-kube-api-access-q8v92\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:13:50.286190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:50.286159    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"
Oct 30 21:13:50.286384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:50.286188    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd" event={"ID":"e201359f-1235-4bf4-a085-b88dc962e9b2","Type":"ContainerDied","Data":"292dcb0c0dc45bc6dd211bcc762c4e6618c15bf3a3546945eab68784b229fa41"}
Oct 30 21:13:50.286384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:50.286240    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="292dcb0c0dc45bc6dd211bcc762c4e6618c15bf3a3546945eab68784b229fa41"
Oct 30 21:13:51.754047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.754007    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"]
Oct 30 21:13:51.755434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.754980    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e201359f-1235-4bf4-a085-b88dc962e9b2" containerName="revoke-keys"
Oct 30 21:13:51.755434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.755003    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e201359f-1235-4bf4-a085-b88dc962e9b2" containerName="revoke-keys"
Oct 30 21:13:51.755434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.755321    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e201359f-1235-4bf4-a085-b88dc962e9b2" containerName="revoke-keys"
Oct 30 21:13:51.784753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.783479    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"]
Oct 30 21:13:51.784753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.783596    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:51.797894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.797822    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:13:51.798279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.798261    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:13:51.798611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.798596    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-v9w8x\""
Oct 30 21:13:51.798848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.798824    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bf44c1ed9483\""
Oct 30 21:13:51.798999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.798985    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:13:51.799235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.799219    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:13:51.898565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.898532    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-routing-overlay-volume\") pod \"payload-processing-external-model-7fc577c689-5rrrw\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") " pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:51.898726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.898602    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-plugins-config-volume\") pod \"payload-processing-external-model-7fc577c689-5rrrw\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") " pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:51.898726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.898698    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-provider-credentials\") pod \"payload-processing-external-model-7fc577c689-5rrrw\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") " pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:51.898843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:51.898817    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spcbv\" (UniqueName: \"kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-kube-api-access-spcbv\") pod \"payload-processing-external-model-7fc577c689-5rrrw\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") " pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:52.028093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:52.009212    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spcbv\" (UniqueName: \"kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-kube-api-access-spcbv\") pod \"payload-processing-external-model-7fc577c689-5rrrw\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") " pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:52.028093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:52.009570    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-routing-overlay-volume\") pod \"payload-processing-external-model-7fc577c689-5rrrw\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") " pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:52.028093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:52.009746    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-plugins-config-volume\") pod \"payload-processing-external-model-7fc577c689-5rrrw\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") " pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:52.028093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:52.009869    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-provider-credentials\") pod \"payload-processing-external-model-7fc577c689-5rrrw\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") " pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:52.028093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:52.011859    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-routing-overlay-volume\") pod \"payload-processing-external-model-7fc577c689-5rrrw\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") " pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:52.028093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:52.018251    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-plugins-config-volume\") pod \"payload-processing-external-model-7fc577c689-5rrrw\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") " pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:52.032891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:52.032864    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-provider-credentials\") pod \"payload-processing-external-model-7fc577c689-5rrrw\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") " pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:52.038195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:52.038167    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spcbv\" (UniqueName: \"kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-kube-api-access-spcbv\") pod \"payload-processing-external-model-7fc577c689-5rrrw\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") " pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:52.106128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:52.106096    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:52.409467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:52.409443    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"]
Oct 30 21:13:52.411272 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:13:52.411239    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a97b65_236c_4c8d_9476_4eea3d8c5ea5.slice/crio-73a99aa42657b21e49b84f19d38eaa8d09eb37bd8cf940e928b545407ad3f18b WatchSource:0}: Error finding container 73a99aa42657b21e49b84f19d38eaa8d09eb37bd8cf940e928b545407ad3f18b: Status 404 returned error can't find the container with id 73a99aa42657b21e49b84f19d38eaa8d09eb37bd8cf940e928b545407ad3f18b
Oct 30 21:13:53.305675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:53.305644    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw" event={"ID":"04a97b65-236c-4c8d-9476-4eea3d8c5ea5","Type":"ContainerStarted","Data":"d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750"}
Oct 30 21:13:53.306096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:53.305694    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw" event={"ID":"04a97b65-236c-4c8d-9476-4eea3d8c5ea5","Type":"ContainerStarted","Data":"73a99aa42657b21e49b84f19d38eaa8d09eb37bd8cf940e928b545407ad3f18b"}
Oct 30 21:13:53.306096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:53.305811    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:13:53.323900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:53.323860    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw" podStartSLOduration=2.323848334 podStartE2EDuration="2.323848334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:53.322266401 +0000 UTC m=+746.368825566" watchObservedRunningTime="2026-10-30 21:13:53.323848334 +0000 UTC m=+746.370407534"
Oct 30 21:13:57.828739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:57.828695    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"]
Oct 30 21:13:57.948734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:57.948694    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"]
Oct 30 21:13:57.950768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:57.950616    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.030881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.030844    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2czgs\" (UniqueName: \"kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-kube-api-access-2czgs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.031044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.030931    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.031044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.030991    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.031171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.031115    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.031214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.031185    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.031260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.031222    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a415d16-9fc3-4b38-9826-1568d98471b4-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.031341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.031324    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.031443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.031421    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.031512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.031454    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.132168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.132063    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.132168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.132128    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.132168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.132144    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.132521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.132217    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2czgs\" (UniqueName: \"kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-kube-api-access-2czgs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.132521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.132272    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.132521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.132293    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.132521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.132333    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.132521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.132367    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.132521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.132439    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a415d16-9fc3-4b38-9826-1568d98471b4-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.132848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.132580    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.133075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.133051    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.133186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.133165    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a415d16-9fc3-4b38-9826-1568d98471b4-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.133298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.133278    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.133458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.133437    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.135579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.135544    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.135700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.135606    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.142709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.142683    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.142945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.142919    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2czgs\" (UniqueName: \"kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-kube-api-access-2czgs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.303914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.303873    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:13:58.662065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.662030    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"]
Oct 30 21:13:58.667419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.667357    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:13:58.667546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.667459    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:13:58.667546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:58.667510    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:13:59.241416 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:13:59.234049    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:13:59.569510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:59.556574    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k" event={"ID":"0a415d16-9fc3-4b38-9826-1568d98471b4","Type":"ContainerStarted","Data":"0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0"}
Oct 30 21:13:59.569510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:59.556643    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k" event={"ID":"0a415d16-9fc3-4b38-9826-1568d98471b4","Type":"ContainerStarted","Data":"af659c7ade804546c177a77dd918f34e3c5de8cea21686b319122c0a90257c19"}
Oct 30 21:13:59.611203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:13:59.611140    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k" podStartSLOduration=2.611118913 podStartE2EDuration="2.611118913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:59.592892066 +0000 UTC m=+752.639451213" watchObservedRunningTime="2026-10-30 21:13:59.611118913 +0000 UTC m=+752.657678059"
Oct 30 21:14:00.318415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:00.317476    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:14:00.332875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:00.331500    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:14:00.332875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:00.331563    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k" podUID="0a415d16-9fc3-4b38-9826-1568d98471b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 21:14:01.310417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:01.305736    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:14:01.310417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:01.305793    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k" podUID="0a415d16-9fc3-4b38-9826-1568d98471b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 21:14:02.308415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:02.307488    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:14:02.308415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:02.307551    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k" podUID="0a415d16-9fc3-4b38-9826-1568d98471b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 21:14:03.310709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:03.310674    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:14:03.311229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:03.310740    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k" podUID="0a415d16-9fc3-4b38-9826-1568d98471b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 21:14:04.309416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:04.308749    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:14:04.309416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:04.308821    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k" podUID="0a415d16-9fc3-4b38-9826-1568d98471b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 21:14:04.315629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:04.315603    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:14:05.304663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:05.304626    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 21:14:05.304827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:05.304678    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k" podUID="0a415d16-9fc3-4b38-9826-1568d98471b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 21:14:06.308815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:06.308789    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:14:06.309242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:06.309223    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:14:06.309965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:06.309944    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:14:17.004477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.003885    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"]
Oct 30 21:14:17.011861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.011832    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.022015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.021984    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3011c6\"/\"e2e-conflict-b-ad08360f-kserve-self-signed-certs\""
Oct 30 21:14:17.023942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.023908    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"]
Oct 30 21:14:17.186614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.186575    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-dshm\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.186822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.186717    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tmp-dir\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.186822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.186788    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmc9g\" (UniqueName: \"kubernetes.io/projected/76b1aadb-6171-4482-b5bf-9c9773b18f3c-kube-api-access-nmc9g\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.186923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.186878    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-home\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.187097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.187066    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tls-certs\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.187309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.187289    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-model-cache\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.290449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.290336    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tls-certs\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.291198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.291176    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-model-cache\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.291835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.291816    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-dshm\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.292414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.291702    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-model-cache\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.292414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.292326    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tmp-dir\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.293175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.292627    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmc9g\" (UniqueName: \"kubernetes.io/projected/76b1aadb-6171-4482-b5bf-9c9773b18f3c-kube-api-access-nmc9g\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.293175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.292667    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tmp-dir\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.293175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.292688    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-home\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.293175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.293137    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-home\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.318522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.318228    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tls-certs\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.345656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.345621    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmc9g\" (UniqueName: \"kubernetes.io/projected/76b1aadb-6171-4482-b5bf-9c9773b18f3c-kube-api-access-nmc9g\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.346632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.346431    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-dshm\") pod \"e2e-conflict-b-ad08360f-kserve-65899779f-jpq97\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") " pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:17.641465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:17.640550    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:18.133506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:18.133477    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"]
Oct 30 21:14:18.813416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:18.812667    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97" event={"ID":"76b1aadb-6171-4482-b5bf-9c9773b18f3c","Type":"ContainerStarted","Data":"8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3"}
Oct 30 21:14:18.813416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:18.812722    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97" event={"ID":"76b1aadb-6171-4482-b5bf-9c9773b18f3c","Type":"ContainerStarted","Data":"3874f0f36c94cbae20cac441f37086909989fdd757a76b386926deb0421fd7f6"}
Oct 30 21:14:19.643519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:19.643484    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:21.290447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:21.290383    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97" podStartSLOduration=5.290368025 podStartE2EDuration="5.290368025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:18.864671376 +0000 UTC m=+771.911230522" watchObservedRunningTime="2026-10-30 21:14:21.290368025 +0000 UTC m=+774.336927170"
Oct 30 21:14:21.290949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:21.290921    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"]
Oct 30 21:14:26.292009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.291941    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw" podUID="04a97b65-236c-4c8d-9476-4eea3d8c5ea5" containerName="payload-processing" containerID="cri-o://d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750" gracePeriod=25
Oct 30 21:14:26.570538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.570509    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:14:26.602572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.602544    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-plugins-config-volume\") pod \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") "
Oct 30 21:14:26.602572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.602576    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-provider-credentials\") pod \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") "
Oct 30 21:14:26.602818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.602739    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-routing-overlay-volume\") pod \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") "
Oct 30 21:14:26.602818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.602783    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-kube-api-access-spcbv\" (UniqueName: \"kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-kube-api-access-spcbv\") pod \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\" (UID: \"04a97b65-236c-4c8d-9476-4eea3d8c5ea5\") "
Oct 30 21:14:26.603104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.603070    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-plugins-config-volume" pod "04a97b65-236c-4c8d-9476-4eea3d8c5ea5" (UID: "04a97b65-236c-4c8d-9476-4eea3d8c5ea5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:26.603219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.603110    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-routing-overlay-volume" pod "04a97b65-236c-4c8d-9476-4eea3d8c5ea5" (UID: "04a97b65-236c-4c8d-9476-4eea3d8c5ea5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:26.603219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.603212    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:26.603349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.603232    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:26.605534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.605504    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-kube-api-access-spcbv" pod "04a97b65-236c-4c8d-9476-4eea3d8c5ea5" (UID: "04a97b65-236c-4c8d-9476-4eea3d8c5ea5"). InnerVolumeSpecName "kube-api-access-spcbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:26.605656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.605587    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-provider-credentials" pod "04a97b65-236c-4c8d-9476-4eea3d8c5ea5" (UID: "04a97b65-236c-4c8d-9476-4eea3d8c5ea5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:26.704677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.704645    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:26.704677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.704675    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spcbv\" (UniqueName: \"kubernetes.io/projected/04a97b65-236c-4c8d-9476-4eea3d8c5ea5-kube-api-access-spcbv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:26.872469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.872309    4918 generic.go:356] "Generic (PLEG): container finished" podID="04a97b65-236c-4c8d-9476-4eea3d8c5ea5" containerID="d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750" exitCode=0
Oct 30 21:14:26.872469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.872372    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"
Oct 30 21:14:26.872469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.872431    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw" event={"ID":"04a97b65-236c-4c8d-9476-4eea3d8c5ea5","Type":"ContainerDied","Data":"d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750"}
Oct 30 21:14:26.872742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.872487    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw" event={"ID":"04a97b65-236c-4c8d-9476-4eea3d8c5ea5","Type":"ContainerDied","Data":"73a99aa42657b21e49b84f19d38eaa8d09eb37bd8cf940e928b545407ad3f18b"}
Oct 30 21:14:26.872742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.872509    4918 scope.go:122] "RemoveContainer" containerID="d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750"
Oct 30 21:14:26.882516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.882497    4918 scope.go:122] "RemoveContainer" containerID="d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750"
Oct 30 21:14:26.882815 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:14:26.882794    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750\": container with ID starting with d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750 not found: ID does not exist" containerID="d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750"
Oct 30 21:14:26.882892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.882824    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750"} err="failed to get container status \"d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750\": rpc error: code = NotFound desc = could not find container \"d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750\": container with ID starting with d079358360e9e739c0476c2bf81d836ad2bf4587f6f7ccb6b658e1c597dec750 not found: ID does not exist"
Oct 30 21:14:26.896165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.896133    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"]
Oct 30 21:14:26.899801 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:26.899780    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fc577c689-5rrrw"]
Oct 30 21:14:27.076977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:27.076944    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a97b65-236c-4c8d-9476-4eea3d8c5ea5" path="/var/lib/kubelet/pods/04a97b65-236c-4c8d-9476-4eea3d8c5ea5/volumes"
Oct 30 21:14:27.641450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:27.641416    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:27.654638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:27.654613    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:27.899304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:27.899227    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:14:29.209250 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:14:29.209214    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:14:29.767859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.767823    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"]
Oct 30 21:14:29.768676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.768648    4918 cpu_manager.go:414] "containerMap: removing container" podUID="04a97b65-236c-4c8d-9476-4eea3d8c5ea5" containerName="payload-processing"
Oct 30 21:14:29.768860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.768840    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a97b65-236c-4c8d-9476-4eea3d8c5ea5" containerName="payload-processing"
Oct 30 21:14:29.769170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.769147    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a97b65-236c-4c8d-9476-4eea3d8c5ea5" containerName="payload-processing"
Oct 30 21:14:29.774562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.774547    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.780320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.780296    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-05a342\"/\"distinct-w2-05a342-kserve-self-signed-certs\""
Oct 30 21:14:29.784480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.784454    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"]
Oct 30 21:14:29.837911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.837882    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-model-cache\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.838039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.837997    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-home\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.838039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.838026    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/970c2613-6a80-4b60-bfce-a51393982944-tls-certs\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.838112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.838074    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-dshm\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.838169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.838156    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xcd4\" (UniqueName: \"kubernetes.io/projected/970c2613-6a80-4b60-bfce-a51393982944-kube-api-access-6xcd4\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.838205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.838196    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-tmp-dir\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.939792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.939724    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-dshm\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.940641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.940587    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xcd4\" (UniqueName: \"kubernetes.io/projected/970c2613-6a80-4b60-bfce-a51393982944-kube-api-access-6xcd4\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.956279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.956237    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-dshm\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.957303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.941476    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-tmp-dir\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.957643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.957622    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-model-cache\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.958050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.958030    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-home\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.958201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.958186    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/970c2613-6a80-4b60-bfce-a51393982944-tls-certs\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.959059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.959040    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-model-cache\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.959497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.959470    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-home\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.966499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.966371    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-tmp-dir\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.979116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.979041    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xcd4\" (UniqueName: \"kubernetes.io/projected/970c2613-6a80-4b60-bfce-a51393982944-kube-api-access-6xcd4\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:29.979861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:29.979835    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/970c2613-6a80-4b60-bfce-a51393982944-tls-certs\") pod \"distinct-w2-05a342-kserve-7cc666b998-dsnpx\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") " pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:30.092327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:30.092247    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:30.369538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:30.369514    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"]
Oct 30 21:14:30.370448 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:14:30.370427    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod970c2613_6a80_4b60_bfce_a51393982944.slice/crio-01a653659c1c367628fbb1d0dace610aa0d0fea190af228b092efe5470d933f2 WatchSource:0}: Error finding container 01a653659c1c367628fbb1d0dace610aa0d0fea190af228b092efe5470d933f2: Status 404 returned error can't find the container with id 01a653659c1c367628fbb1d0dace610aa0d0fea190af228b092efe5470d933f2
Oct 30 21:14:30.949329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:30.949290    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" event={"ID":"970c2613-6a80-4b60-bfce-a51393982944","Type":"ContainerStarted","Data":"614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7"}
Oct 30 21:14:30.949548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:30.949336    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" event={"ID":"970c2613-6a80-4b60-bfce-a51393982944","Type":"ContainerStarted","Data":"01a653659c1c367628fbb1d0dace610aa0d0fea190af228b092efe5470d933f2"}
Oct 30 21:14:30.971916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:30.971861    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podStartSLOduration=1.9718474019999999 podStartE2EDuration="1.971847402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:30.969484936 +0000 UTC m=+784.016044102" watchObservedRunningTime="2026-10-30 21:14:30.971847402 +0000 UTC m=+784.018406547"
Oct 30 21:14:31.104417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:31.093157    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:40.093205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:40.093166    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:40.106029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:40.106004    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:41.081428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:41.081138    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:14:46.826063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:46.826034    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"]
Oct 30 21:14:51.268984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.268945    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"]
Oct 30 21:14:51.273439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.273421    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:51.281421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.281368    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:14:51.281549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.281471    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:14:51.281549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.281492    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:14:51.281549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.281536    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:14:51.284214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.284193    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"]
Oct 30 21:14:51.389700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.389664    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0fbd348-66v4s\" (UID: \"25659f1b-4552-4454-996f-58b149dfb0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:51.389700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.389702    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0fbd348-66v4s\" (UID: \"25659f1b-4552-4454-996f-58b149dfb0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:51.389918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.389869    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqtmr\" (UniqueName: \"kubernetes.io/projected/25659f1b-4552-4454-996f-58b149dfb0a7-kube-api-access-rqtmr\") pod \"maas-api-revoke-sub-a0fbd348-66v4s\" (UID: \"25659f1b-4552-4454-996f-58b149dfb0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:51.490955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.490914    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0fbd348-66v4s\" (UID: \"25659f1b-4552-4454-996f-58b149dfb0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:51.490955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.490954    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0fbd348-66v4s\" (UID: \"25659f1b-4552-4454-996f-58b149dfb0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:51.491177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.491045    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqtmr\" (UniqueName: \"kubernetes.io/projected/25659f1b-4552-4454-996f-58b149dfb0a7-kube-api-access-rqtmr\") pod \"maas-api-revoke-sub-a0fbd348-66v4s\" (UID: \"25659f1b-4552-4454-996f-58b149dfb0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:51.491760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.491740    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0fbd348-66v4s\" (UID: \"25659f1b-4552-4454-996f-58b149dfb0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:51.493520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.493500    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0fbd348-66v4s\" (UID: \"25659f1b-4552-4454-996f-58b149dfb0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:51.502288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.502267    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqtmr\" (UniqueName: \"kubernetes.io/projected/25659f1b-4552-4454-996f-58b149dfb0a7-kube-api-access-rqtmr\") pod \"maas-api-revoke-sub-a0fbd348-66v4s\" (UID: \"25659f1b-4552-4454-996f-58b149dfb0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:51.591667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.591587    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:51.906458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:51.906182    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"]
Oct 30 21:14:52.228499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:52.228459    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s" event={"ID":"25659f1b-4552-4454-996f-58b149dfb0a7","Type":"ContainerStarted","Data":"a96893ad7dc069f54f4452f2ea32988088005ed01b7eb90fc048b7f219611855"}
Oct 30 21:14:52.228681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:52.228517    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s" event={"ID":"25659f1b-4552-4454-996f-58b149dfb0a7","Type":"ContainerStarted","Data":"ff02db438fd87b7e5fcd8851353f919a84f12b495e268e0b26be78269d4e46c6"}
Oct 30 21:14:52.247660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:52.247613    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s" podStartSLOduration=1.247596068 podStartE2EDuration="1.247596068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:52.244634766 +0000 UTC m=+805.291193926" watchObservedRunningTime="2026-10-30 21:14:52.247596068 +0000 UTC m=+805.294155214"
Oct 30 21:14:53.241470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:53.241438    4918 generic.go:356] "Generic (PLEG): container finished" podID="25659f1b-4552-4454-996f-58b149dfb0a7" containerID="a96893ad7dc069f54f4452f2ea32988088005ed01b7eb90fc048b7f219611855" exitCode=0
Oct 30 21:14:53.241863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:53.241528    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s" event={"ID":"25659f1b-4552-4454-996f-58b149dfb0a7","Type":"ContainerDied","Data":"a96893ad7dc069f54f4452f2ea32988088005ed01b7eb90fc048b7f219611855"}
Oct 30 21:14:54.452333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:54.452307    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:54.539161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:54.539090    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25659f1b-4552-4454-996f-58b149dfb0a7-kube-api-access-rqtmr\" (UniqueName: \"kubernetes.io/projected/25659f1b-4552-4454-996f-58b149dfb0a7-kube-api-access-rqtmr\") pod \"25659f1b-4552-4454-996f-58b149dfb0a7\" (UID: \"25659f1b-4552-4454-996f-58b149dfb0a7\") "
Oct 30 21:14:54.539161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:54.539154    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-xks-ca\") pod \"25659f1b-4552-4454-996f-58b149dfb0a7\" (UID: \"25659f1b-4552-4454-996f-58b149dfb0a7\") "
Oct 30 21:14:54.539364 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:54.539226    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-openshift-ca\") pod \"25659f1b-4552-4454-996f-58b149dfb0a7\" (UID: \"25659f1b-4552-4454-996f-58b149dfb0a7\") "
Oct 30 21:14:54.539641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:54.539617    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-openshift-ca" pod "25659f1b-4552-4454-996f-58b149dfb0a7" (UID: "25659f1b-4552-4454-996f-58b149dfb0a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:14:54.541361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:54.541337    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25659f1b-4552-4454-996f-58b149dfb0a7-kube-api-access-rqtmr" pod "25659f1b-4552-4454-996f-58b149dfb0a7" (UID: "25659f1b-4552-4454-996f-58b149dfb0a7"). InnerVolumeSpecName "kube-api-access-rqtmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:14:54.541452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:54.541343    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-xks-ca" pod "25659f1b-4552-4454-996f-58b149dfb0a7" (UID: "25659f1b-4552-4454-996f-58b149dfb0a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:14:54.640952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:54.640917    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:54.640952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:54.640946    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqtmr\" (UniqueName: \"kubernetes.io/projected/25659f1b-4552-4454-996f-58b149dfb0a7-kube-api-access-rqtmr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:54.640952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:54.640959    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25659f1b-4552-4454-996f-58b149dfb0a7-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:14:55.256026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:55.256003    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"
Oct 30 21:14:55.256189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:55.256041    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s" event={"ID":"25659f1b-4552-4454-996f-58b149dfb0a7","Type":"ContainerDied","Data":"ff02db438fd87b7e5fcd8851353f919a84f12b495e268e0b26be78269d4e46c6"}
Oct 30 21:14:55.256189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:55.256083    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff02db438fd87b7e5fcd8851353f919a84f12b495e268e0b26be78269d4e46c6"
Oct 30 21:14:58.866272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:58.866239    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"]
Oct 30 21:14:58.867027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:58.867002    4918 cpu_manager.go:414] "containerMap: removing container" podUID="25659f1b-4552-4454-996f-58b149dfb0a7" containerName="revoke-keys"
Oct 30 21:14:58.867027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:58.867026    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25659f1b-4552-4454-996f-58b149dfb0a7" containerName="revoke-keys"
Oct 30 21:14:58.867258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:58.867243    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="25659f1b-4552-4454-996f-58b149dfb0a7" containerName="revoke-keys"
Oct 30 21:14:58.871286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:58.871266    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:58.880149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:58.880123    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3011c6\"/\"distinct-w4-3011c6-kserve-self-signed-certs\""
Oct 30 21:14:58.882639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:58.882619    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"]
Oct 30 21:14:59.021424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.021361    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-home\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.021741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.021716    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-dshm\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.021879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.021861    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ff176b-aa57-4514-a607-0b1513ca3bb0-tls-certs\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.022033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.022015    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54gg\" (UniqueName: \"kubernetes.io/projected/08ff176b-aa57-4514-a607-0b1513ca3bb0-kube-api-access-z54gg\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.022244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.022230    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-model-cache\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.022438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.022421    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-tmp-dir\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.123453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.123343    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-model-cache\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.123453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.123445    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-tmp-dir\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.123690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.123506    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-home\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.123690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.123552    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-dshm\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.123690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.123567    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ff176b-aa57-4514-a607-0b1513ca3bb0-tls-certs\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.123690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.123623    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54gg\" (UniqueName: \"kubernetes.io/projected/08ff176b-aa57-4514-a607-0b1513ca3bb0-kube-api-access-z54gg\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.123903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.123800    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-model-cache\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.123903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.123812    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-tmp-dir\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.124016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.123942    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-home\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.126005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.125976    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-dshm\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.126238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.126223    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ff176b-aa57-4514-a607-0b1513ca3bb0-tls-certs\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.136449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.136424    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54gg\" (UniqueName: \"kubernetes.io/projected/08ff176b-aa57-4514-a607-0b1513ca3bb0-kube-api-access-z54gg\") pod \"distinct-w4-3011c6-kserve-65785b867c-vc6pn\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.161831 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:14:59.161788    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:14:59.186145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.186119    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:14:59.467879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:14:59.467851    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"]
Oct 30 21:14:59.469902 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:14:59.469854    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08ff176b_aa57_4514_a607_0b1513ca3bb0.slice/crio-5340b661f2573963b090b7ef7ffa6940fcf4a1132728ba72e7a09be460de002d WatchSource:0}: Error finding container 5340b661f2573963b090b7ef7ffa6940fcf4a1132728ba72e7a09be460de002d: Status 404 returned error can't find the container with id 5340b661f2573963b090b7ef7ffa6940fcf4a1132728ba72e7a09be460de002d
Oct 30 21:15:00.157870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.157839    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"]
Oct 30 21:15:00.161771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.161755    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"]
Oct 30 21:15:00.161894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.161874    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"
Oct 30 21:15:00.165410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.165375    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"]
Oct 30 21:15:00.165529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.165479    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"
Oct 30 21:15:00.168860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.168843    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"]
Oct 30 21:15:00.168978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.168953    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"
Oct 30 21:15:00.172069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.172040    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:15:00.172232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.172204    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:15:00.172492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.172349    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"
Oct 30 21:15:00.176224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.176194    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:15:00.177211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.177194    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"]
Oct 30 21:15:00.181165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.181141    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"]
Oct 30 21:15:00.184540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.184084    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"]
Oct 30 21:15:00.210288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.210257    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"]
Oct 30 21:15:00.296700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.296661    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" event={"ID":"08ff176b-aa57-4514-a607-0b1513ca3bb0","Type":"ContainerStarted","Data":"c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64"}
Oct 30 21:15:00.296856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.296714    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" event={"ID":"08ff176b-aa57-4514-a607-0b1513ca3bb0","Type":"ContainerStarted","Data":"5340b661f2573963b090b7ef7ffa6940fcf4a1132728ba72e7a09be460de002d"}
Oct 30 21:15:00.321945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.321884    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podStartSLOduration=2.321871622 podStartE2EDuration="2.321871622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:00.319809639 +0000 UTC m=+813.366368785" watchObservedRunningTime="2026-10-30 21:15:00.321871622 +0000 UTC m=+813.368430767"
Oct 30 21:15:00.336628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.336599    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bqf7\" (UniqueName: \"kubernetes.io/projected/681c66ea-6c52-45ac-a2b5-6e4ea80188be-kube-api-access-9bqf7\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d\" (UID: \"681c66ea-6c52-45ac-a2b5-6e4ea80188be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"
Oct 30 21:15:00.336755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.336669    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681c66ea-6c52-45ac-a2b5-6e4ea80188be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d\" (UID: \"681c66ea-6c52-45ac-a2b5-6e4ea80188be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"
Oct 30 21:15:00.336809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.336769    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jpzz\" (UniqueName: \"kubernetes.io/projected/3615fef6-3ddb-47ba-b359-a8431b57d513-kube-api-access-8jpzz\") pod \"maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt\" (UID: \"3615fef6-3ddb-47ba-b359-a8431b57d513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"
Oct 30 21:15:00.336846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.336816    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-7l7f5\" (UID: \"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"
Oct 30 21:15:00.336937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.336920    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xghzw\" (UniqueName: \"kubernetes.io/projected/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-kube-api-access-xghzw\") pod \"maas-api-key-cleanup-29889915-7l7f5\" (UID: \"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"
Oct 30 21:15:00.337034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.336975    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3615fef6-3ddb-47ba-b359-a8431b57d513-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt\" (UID: \"3615fef6-3ddb-47ba-b359-a8431b57d513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"
Oct 30 21:15:00.337101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.337085    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntw7\" (UniqueName: \"kubernetes.io/projected/71b6f57f-52a0-460c-87ae-db559858428b-kube-api-access-5ntw7\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p\" (UID: \"71b6f57f-52a0-460c-87ae-db559858428b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"
Oct 30 21:15:00.337155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.337109    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b6f57f-52a0-460c-87ae-db559858428b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p\" (UID: \"71b6f57f-52a0-460c-87ae-db559858428b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"
Oct 30 21:15:00.443322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.438473    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xghzw\" (UniqueName: \"kubernetes.io/projected/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-kube-api-access-xghzw\") pod \"maas-api-key-cleanup-29889915-7l7f5\" (UID: \"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"
Oct 30 21:15:00.443322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.438561    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3615fef6-3ddb-47ba-b359-a8431b57d513-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt\" (UID: \"3615fef6-3ddb-47ba-b359-a8431b57d513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"
Oct 30 21:15:00.443322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.438666    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntw7\" (UniqueName: \"kubernetes.io/projected/71b6f57f-52a0-460c-87ae-db559858428b-kube-api-access-5ntw7\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p\" (UID: \"71b6f57f-52a0-460c-87ae-db559858428b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"
Oct 30 21:15:00.443322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.438698    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b6f57f-52a0-460c-87ae-db559858428b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p\" (UID: \"71b6f57f-52a0-460c-87ae-db559858428b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"
Oct 30 21:15:00.443322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.438775    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bqf7\" (UniqueName: \"kubernetes.io/projected/681c66ea-6c52-45ac-a2b5-6e4ea80188be-kube-api-access-9bqf7\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d\" (UID: \"681c66ea-6c52-45ac-a2b5-6e4ea80188be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"
Oct 30 21:15:00.443322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.438882    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681c66ea-6c52-45ac-a2b5-6e4ea80188be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d\" (UID: \"681c66ea-6c52-45ac-a2b5-6e4ea80188be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"
Oct 30 21:15:00.443322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.438945    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jpzz\" (UniqueName: \"kubernetes.io/projected/3615fef6-3ddb-47ba-b359-a8431b57d513-kube-api-access-8jpzz\") pod \"maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt\" (UID: \"3615fef6-3ddb-47ba-b359-a8431b57d513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"
Oct 30 21:15:00.443322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.439002    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-7l7f5\" (UID: \"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"
Oct 30 21:15:00.443322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.439970    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889915-7l7f5\" (UID: \"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"
Oct 30 21:15:00.475433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.466757    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xghzw\" (UniqueName: \"kubernetes.io/projected/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-kube-api-access-xghzw\") pod \"maas-api-key-cleanup-29889915-7l7f5\" (UID: \"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"
Oct 30 21:15:00.475433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.468343    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681c66ea-6c52-45ac-a2b5-6e4ea80188be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d\" (UID: \"681c66ea-6c52-45ac-a2b5-6e4ea80188be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"
Oct 30 21:15:00.476371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.476335    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3615fef6-3ddb-47ba-b359-a8431b57d513-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt\" (UID: \"3615fef6-3ddb-47ba-b359-a8431b57d513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"
Oct 30 21:15:00.495421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.489188    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b6f57f-52a0-460c-87ae-db559858428b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p\" (UID: \"71b6f57f-52a0-460c-87ae-db559858428b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"
Oct 30 21:15:00.499432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.499289    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jpzz\" (UniqueName: \"kubernetes.io/projected/3615fef6-3ddb-47ba-b359-a8431b57d513-kube-api-access-8jpzz\") pod \"maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt\" (UID: \"3615fef6-3ddb-47ba-b359-a8431b57d513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"
Oct 30 21:15:00.507413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.501762    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntw7\" (UniqueName: \"kubernetes.io/projected/71b6f57f-52a0-460c-87ae-db559858428b-kube-api-access-5ntw7\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p\" (UID: \"71b6f57f-52a0-460c-87ae-db559858428b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"
Oct 30 21:15:00.507413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.502371    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bqf7\" (UniqueName: \"kubernetes.io/projected/681c66ea-6c52-45ac-a2b5-6e4ea80188be-kube-api-access-9bqf7\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d\" (UID: \"681c66ea-6c52-45ac-a2b5-6e4ea80188be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"
Oct 30 21:15:00.526451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.511239    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"
Oct 30 21:15:00.527443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.527219    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"
Oct 30 21:15:00.783553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.783523    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"
Oct 30 21:15:00.794679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.794652    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"
Oct 30 21:15:00.822607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.822546    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"]
Oct 30 21:15:00.825045 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:00.825014    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb9b6b2_bcbf_4439_b88c_32ccba5878cf.slice/crio-77c1aa475506a62f95d017c506d05de9ede2a829e08a37e20a4c74ba5fb4615a WatchSource:0}: Error finding container 77c1aa475506a62f95d017c506d05de9ede2a829e08a37e20a4c74ba5fb4615a: Status 404 returned error can't find the container with id 77c1aa475506a62f95d017c506d05de9ede2a829e08a37e20a4c74ba5fb4615a
Oct 30 21:15:00.826059 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:00.826027    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3615fef6_3ddb_47ba_b359_a8431b57d513.slice/crio-1a863577f13a595748caa035153a6f7a91295628b13653ec02fadae60cfa1553 WatchSource:0}: Error finding container 1a863577f13a595748caa035153a6f7a91295628b13653ec02fadae60cfa1553: Status 404 returned error can't find the container with id 1a863577f13a595748caa035153a6f7a91295628b13653ec02fadae60cfa1553
Oct 30 21:15:00.828146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:00.827893    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"]
Oct 30 21:15:01.071019 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:01.070990    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b6f57f_52a0_460c_87ae_db559858428b.slice/crio-f9e28941fa763fc92a900578032c33246042c5ab1c2b93749b7ca5059d58b855 WatchSource:0}: Error finding container f9e28941fa763fc92a900578032c33246042c5ab1c2b93749b7ca5059d58b855: Status 404 returned error can't find the container with id f9e28941fa763fc92a900578032c33246042c5ab1c2b93749b7ca5059d58b855
Oct 30 21:15:01.080202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.080174    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"]
Oct 30 21:15:01.132384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.131036    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"]
Oct 30 21:15:01.186564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.186534    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:15:01.305416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.305356    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5" event={"ID":"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf","Type":"ContainerStarted","Data":"9ad915f5fc7306b37a176a20ba027474194ad04292060b2ea425919dacb74fa8"}
Oct 30 21:15:01.305538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.305444    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5" event={"ID":"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf","Type":"ContainerStarted","Data":"77c1aa475506a62f95d017c506d05de9ede2a829e08a37e20a4c74ba5fb4615a"}
Oct 30 21:15:01.307050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.307026    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt" event={"ID":"3615fef6-3ddb-47ba-b359-a8431b57d513","Type":"ContainerStarted","Data":"2727cb692ece0f42afac8097b9be398330056e720595397ce74e15a1331730eb"}
Oct 30 21:15:01.307159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.307075    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt" event={"ID":"3615fef6-3ddb-47ba-b359-a8431b57d513","Type":"ContainerStarted","Data":"1a863577f13a595748caa035153a6f7a91295628b13653ec02fadae60cfa1553"}
Oct 30 21:15:01.308338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.308314    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p" event={"ID":"71b6f57f-52a0-460c-87ae-db559858428b","Type":"ContainerStarted","Data":"f9e28941fa763fc92a900578032c33246042c5ab1c2b93749b7ca5059d58b855"}
Oct 30 21:15:01.309683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.309656    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d" event={"ID":"681c66ea-6c52-45ac-a2b5-6e4ea80188be","Type":"ContainerStarted","Data":"2681d916b33b520226ad69752ca5431851c69b546ba0063d9f8db6b59eca8183"}
Oct 30 21:15:01.324499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.324415    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5" podStartSLOduration=1.32438179 podStartE2EDuration="1.32438179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:01.321726531 +0000 UTC m=+814.368285687" watchObservedRunningTime="2026-10-30 21:15:01.32438179 +0000 UTC m=+814.370940939"
Oct 30 21:15:01.340423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.340339    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt" podStartSLOduration=1.3403275080000001 podStartE2EDuration="1.340327508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:01.338548331 +0000 UTC m=+814.385107475" watchObservedRunningTime="2026-10-30 21:15:01.340327508 +0000 UTC m=+814.386886652"
Oct 30 21:15:01.872491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.869473    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97" podUID="76b1aadb-6171-4482-b5bf-9c9773b18f3c" containerName="main" containerID="cri-o://8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3" gracePeriod=45
Oct 30 21:15:01.888722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:01.888686    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97" podUID="76b1aadb-6171-4482-b5bf-9c9773b18f3c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": dial tcp 10.132.0.79:8000: connect: connection refused"
Oct 30 21:15:02.318724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.318690    4918 generic.go:356] "Generic (PLEG): container finished" podID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" containerID="9ad915f5fc7306b37a176a20ba027474194ad04292060b2ea425919dacb74fa8" exitCode=60
Oct 30 21:15:02.319155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.318776    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5" event={"ID":"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf","Type":"ContainerDied","Data":"9ad915f5fc7306b37a176a20ba027474194ad04292060b2ea425919dacb74fa8"}
Oct 30 21:15:02.319155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.319112    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:15:02.319155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.319130    4918 scope.go:122] "RemoveContainer" containerID="9ad915f5fc7306b37a176a20ba027474194ad04292060b2ea425919dacb74fa8"
Oct 30 21:15:02.320144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.320125    4918 generic.go:356] "Generic (PLEG): container finished" podID="76b1aadb-6171-4482-b5bf-9c9773b18f3c" containerID="8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3" exitCode=0
Oct 30 21:15:02.320242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.320198    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97" event={"ID":"76b1aadb-6171-4482-b5bf-9c9773b18f3c","Type":"ContainerDied","Data":"8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3"}
Oct 30 21:15:02.320306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.320239    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97" event={"ID":"76b1aadb-6171-4482-b5bf-9c9773b18f3c","Type":"ContainerDied","Data":"3874f0f36c94cbae20cac441f37086909989fdd757a76b386926deb0421fd7f6"}
Oct 30 21:15:02.320306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.320262    4918 scope.go:122] "RemoveContainer" containerID="8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3"
Oct 30 21:15:02.321974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.321954    4918 generic.go:356] "Generic (PLEG): container finished" podID="3615fef6-3ddb-47ba-b359-a8431b57d513" containerID="2727cb692ece0f42afac8097b9be398330056e720595397ce74e15a1331730eb" exitCode=60
Oct 30 21:15:02.322120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.322104    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt" event={"ID":"3615fef6-3ddb-47ba-b359-a8431b57d513","Type":"ContainerDied","Data":"2727cb692ece0f42afac8097b9be398330056e720595397ce74e15a1331730eb"}
Oct 30 21:15:02.322532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.322270    4918 scope.go:122] "RemoveContainer" containerID="2727cb692ece0f42afac8097b9be398330056e720595397ce74e15a1331730eb"
Oct 30 21:15:02.324023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.324000    4918 generic.go:356] "Generic (PLEG): container finished" podID="71b6f57f-52a0-460c-87ae-db559858428b" containerID="4d32cc36bd04cec47a9305e8c1aa403a5c6812006a63615f871bd16c3bbe6721" exitCode=60
Oct 30 21:15:02.324133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.324078    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p" event={"ID":"71b6f57f-52a0-460c-87ae-db559858428b","Type":"ContainerDied","Data":"4d32cc36bd04cec47a9305e8c1aa403a5c6812006a63615f871bd16c3bbe6721"}
Oct 30 21:15:02.324327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.324313    4918 scope.go:122] "RemoveContainer" containerID="4d32cc36bd04cec47a9305e8c1aa403a5c6812006a63615f871bd16c3bbe6721"
Oct 30 21:15:02.327445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.326293    4918 generic.go:356] "Generic (PLEG): container finished" podID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" containerID="8ecc1b76a62409f0ba9976d867ec6fdf0982677f3bec1122c9b7c30d3f2d3f34" exitCode=60
Oct 30 21:15:02.327445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.327106    4918 scope.go:122] "RemoveContainer" containerID="8ecc1b76a62409f0ba9976d867ec6fdf0982677f3bec1122c9b7c30d3f2d3f34"
Oct 30 21:15:02.327445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.327372    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d" event={"ID":"681c66ea-6c52-45ac-a2b5-6e4ea80188be","Type":"ContainerDied","Data":"8ecc1b76a62409f0ba9976d867ec6fdf0982677f3bec1122c9b7c30d3f2d3f34"}
Oct 30 21:15:02.338619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.338596    4918 scope.go:122] "RemoveContainer" containerID="8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3"
Oct 30 21:15:02.338936 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:02.338905    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3\": container with ID starting with 8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3 not found: ID does not exist" containerID="8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3"
Oct 30 21:15:02.339026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.338948    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3"} err="failed to get container status \"8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3\": rpc error: code = NotFound desc = could not find container \"8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3\": container with ID starting with 8c3e635c768a573b71218d64304564cb9dfccfaff36ebcdc8978304281dbb5f3 not found: ID does not exist"
Oct 30 21:15:02.478759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.478730    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b1aadb-6171-4482-b5bf-9c9773b18f3c-kube-api-access-nmc9g\" (UniqueName: \"kubernetes.io/projected/76b1aadb-6171-4482-b5bf-9c9773b18f3c-kube-api-access-nmc9g\") pod \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") "
Oct 30 21:15:02.478902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.478818    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-model-cache\") pod \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") "
Oct 30 21:15:02.478902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.478850    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-home\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-home\") pod \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") "
Oct 30 21:15:02.478990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.478931    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tmp-dir\") pod \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") "
Oct 30 21:15:02.479056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.479041    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-dshm\") pod \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") "
Oct 30 21:15:02.479109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.479074    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tls-certs\") pod \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\" (UID: \"76b1aadb-6171-4482-b5bf-9c9773b18f3c\") "
Oct 30 21:15:02.479109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.479072    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-model-cache" pod "76b1aadb-6171-4482-b5bf-9c9773b18f3c" (UID: "76b1aadb-6171-4482-b5bf-9c9773b18f3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:02.479207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.479133    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-home" pod "76b1aadb-6171-4482-b5bf-9c9773b18f3c" (UID: "76b1aadb-6171-4482-b5bf-9c9773b18f3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:02.479269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.479238    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tmp-dir" pod "76b1aadb-6171-4482-b5bf-9c9773b18f3c" (UID: "76b1aadb-6171-4482-b5bf-9c9773b18f3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:02.479849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.479750    4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:02.479849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.479779    4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:02.479849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.479795    4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:02.482138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.482112    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tls-certs" pod "76b1aadb-6171-4482-b5bf-9c9773b18f3c" (UID: "76b1aadb-6171-4482-b5bf-9c9773b18f3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:02.482138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.482121    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-dshm" pod "76b1aadb-6171-4482-b5bf-9c9773b18f3c" (UID: "76b1aadb-6171-4482-b5bf-9c9773b18f3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:15:02.482304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.482199    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b1aadb-6171-4482-b5bf-9c9773b18f3c-kube-api-access-nmc9g" pod "76b1aadb-6171-4482-b5bf-9c9773b18f3c" (UID: "76b1aadb-6171-4482-b5bf-9c9773b18f3c"). InnerVolumeSpecName "kube-api-access-nmc9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:02.581155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.581086    4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b1aadb-6171-4482-b5bf-9c9773b18f3c-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:02.581155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.581111    4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b1aadb-6171-4482-b5bf-9c9773b18f3c-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:02.581155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.581122    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmc9g\" (UniqueName: \"kubernetes.io/projected/76b1aadb-6171-4482-b5bf-9c9773b18f3c-kube-api-access-nmc9g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:02.746921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.724458    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-698f9c649-d76kx"]
Oct 30 21:15:02.746921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.725309    4918 cpu_manager.go:414] "containerMap: removing container" podUID="76b1aadb-6171-4482-b5bf-9c9773b18f3c" containerName="main"
Oct 30 21:15:02.746921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.725327    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b1aadb-6171-4482-b5bf-9c9773b18f3c" containerName="main"
Oct 30 21:15:02.766949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.766894    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b1aadb-6171-4482-b5bf-9c9773b18f3c" containerName="main"
Oct 30 21:15:02.788981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.788835    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-698f9c649-d76kx"]
Oct 30 21:15:02.790598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.788965    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:02.796767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.796736    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:15:02.797681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.797261    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:15:02.797681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.797342    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:15:02.797681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.797561    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:15:02.797681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.796751    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-22787fe12174\""
Oct 30 21:15:02.797681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.797624    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-v9w8x\""
Oct 30 21:15:02.889184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.889089    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-provider-credentials\") pod \"payload-processing-external-model-698f9c649-d76kx\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:02.889184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.889151    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbd84\" (UniqueName: \"kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-kube-api-access-tbd84\") pod \"payload-processing-external-model-698f9c649-d76kx\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:02.889449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.889296    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-plugins-config-volume\") pod \"payload-processing-external-model-698f9c649-d76kx\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:02.889449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:02.889346    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-routing-overlay-volume\") pod \"payload-processing-external-model-698f9c649-d76kx\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:03.000801 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.000230    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-provider-credentials\") pod \"payload-processing-external-model-698f9c649-d76kx\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:03.001513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.001116    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbd84\" (UniqueName: \"kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-kube-api-access-tbd84\") pod \"payload-processing-external-model-698f9c649-d76kx\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:03.002721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.001729    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-plugins-config-volume\") pod \"payload-processing-external-model-698f9c649-d76kx\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:03.002721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.002682    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-plugins-config-volume\") pod \"payload-processing-external-model-698f9c649-d76kx\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:03.003826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.002943    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-routing-overlay-volume\") pod \"payload-processing-external-model-698f9c649-d76kx\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:03.003826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.003783    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-routing-overlay-volume\") pod \"payload-processing-external-model-698f9c649-d76kx\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:03.010144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.010089    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-provider-credentials\") pod \"payload-processing-external-model-698f9c649-d76kx\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:03.032266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.032195    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbd84\" (UniqueName: \"kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-kube-api-access-tbd84\") pod \"payload-processing-external-model-698f9c649-d76kx\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") " pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:03.114657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.114634    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:03.336609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.336580    4918 generic.go:356] "Generic (PLEG): container finished" podID="71b6f57f-52a0-460c-87ae-db559858428b" containerID="8eea5367f407e9ecb96e8ea301ba13a218124be7f911ed163dbbbf1b1a7fd309" exitCode=60
Oct 30 21:15:03.337036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.336658    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p" event={"ID":"71b6f57f-52a0-460c-87ae-db559858428b","Type":"ContainerDied","Data":"8eea5367f407e9ecb96e8ea301ba13a218124be7f911ed163dbbbf1b1a7fd309"}
Oct 30 21:15:03.337036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.336717    4918 scope.go:122] "RemoveContainer" containerID="4d32cc36bd04cec47a9305e8c1aa403a5c6812006a63615f871bd16c3bbe6721"
Oct 30 21:15:03.337036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.336973    4918 scope.go:122] "RemoveContainer" containerID="8eea5367f407e9ecb96e8ea301ba13a218124be7f911ed163dbbbf1b1a7fd309"
Oct 30 21:15:03.337296 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:03.337273    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p_odh-ai-gateway-infra(71b6f57f-52a0-460c-87ae-db559858428b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p" podUID="71b6f57f-52a0-460c-87ae-db559858428b"
Oct 30 21:15:03.338782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.338758    4918 generic.go:356] "Generic (PLEG): container finished" podID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" containerID="e6574ddd543984063877aa3961dedd10b1d80feb8a1ff130158c8458c3ca0d72" exitCode=60
Oct 30 21:15:03.338872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.338833    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d" event={"ID":"681c66ea-6c52-45ac-a2b5-6e4ea80188be","Type":"ContainerDied","Data":"e6574ddd543984063877aa3961dedd10b1d80feb8a1ff130158c8458c3ca0d72"}
Oct 30 21:15:03.339106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.339092    4918 scope.go:122] "RemoveContainer" containerID="e6574ddd543984063877aa3961dedd10b1d80feb8a1ff130158c8458c3ca0d72"
Oct 30 21:15:03.339406 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:03.339366    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d_odh-ai-gateway-infra(681c66ea-6c52-45ac-a2b5-6e4ea80188be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be"
Oct 30 21:15:03.341049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.341023    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5" event={"ID":"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf","Type":"ContainerStarted","Data":"3008b33f705783db731ea00ba92998ece4666aca6837411f19f81821749aba7e"}
Oct 30 21:15:03.342092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.342076    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"
Oct 30 21:15:03.344118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.344100    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt" event={"ID":"3615fef6-3ddb-47ba-b359-a8431b57d513","Type":"ContainerStarted","Data":"4b59c0d57078335f9c5b6cc1ef21edee88ff43a18979af25f90ccbd984d6e417"}
Oct 30 21:15:03.393303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.393243    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-698f9c649-d76kx"]
Oct 30 21:15:03.419522 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:03.419490    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9711c192_fc19_4c13_8615_1b3d3bb945df.slice/crio-dcad2ab290f77cbc0ecc7964246ce012944b6989225c363e7fdd50910cc3757c WatchSource:0}: Error finding container dcad2ab290f77cbc0ecc7964246ce012944b6989225c363e7fdd50910cc3757c: Status 404 returned error can't find the container with id dcad2ab290f77cbc0ecc7964246ce012944b6989225c363e7fdd50910cc3757c
Oct 30 21:15:03.526725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.526702    4918 scope.go:122] "RemoveContainer" containerID="8ecc1b76a62409f0ba9976d867ec6fdf0982677f3bec1122c9b7c30d3f2d3f34"
Oct 30 21:15:03.551559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.551536    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"]
Oct 30 21:15:03.564996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.564957    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/e2e-conflict-b-ad08360f-kserve-65899779f-jpq97"]
Oct 30 21:15:03.568652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.568626    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"]
Oct 30 21:15:03.572892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.572839    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:03.573624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.573597    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"]
Oct 30 21:15:03.577342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.577321    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:03.611098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.611069    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a48a8269-sgscm\" (UID: \"d9dbe6fd-1277-4644-9906-0fb0797bb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:03.611294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.611265    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a48a8269-sgscm\" (UID: \"d9dbe6fd-1277-4644-9906-0fb0797bb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:03.611384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.611366    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpwmp\" (UniqueName: \"kubernetes.io/projected/d9dbe6fd-1277-4644-9906-0fb0797bb199-kube-api-access-cpwmp\") pod \"maas-api-revoke-sub-a48a8269-sgscm\" (UID: \"d9dbe6fd-1277-4644-9906-0fb0797bb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:03.712154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.712065    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a48a8269-sgscm\" (UID: \"d9dbe6fd-1277-4644-9906-0fb0797bb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:03.712154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.712125    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpwmp\" (UniqueName: \"kubernetes.io/projected/d9dbe6fd-1277-4644-9906-0fb0797bb199-kube-api-access-cpwmp\") pod \"maas-api-revoke-sub-a48a8269-sgscm\" (UID: \"d9dbe6fd-1277-4644-9906-0fb0797bb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:03.712355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.712172    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a48a8269-sgscm\" (UID: \"d9dbe6fd-1277-4644-9906-0fb0797bb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:03.712936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.712873    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a48a8269-sgscm\" (UID: \"d9dbe6fd-1277-4644-9906-0fb0797bb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:03.714676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.714655    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a48a8269-sgscm\" (UID: \"d9dbe6fd-1277-4644-9906-0fb0797bb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:03.723257 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.723232    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpwmp\" (UniqueName: \"kubernetes.io/projected/d9dbe6fd-1277-4644-9906-0fb0797bb199-kube-api-access-cpwmp\") pod \"maas-api-revoke-sub-a48a8269-sgscm\" (UID: \"d9dbe6fd-1277-4644-9906-0fb0797bb199\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:03.890664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:03.890635    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:04.165940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.165913    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"]
Oct 30 21:15:04.166759 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:04.166736    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9dbe6fd_1277_4644_9906_0fb0797bb199.slice/crio-c8436c929a6e76a2d21a71694b2a31ee2f339aa5559570266c23fdd39f6fb3d7 WatchSource:0}: Error finding container c8436c929a6e76a2d21a71694b2a31ee2f339aa5559570266c23fdd39f6fb3d7: Status 404 returned error can't find the container with id c8436c929a6e76a2d21a71694b2a31ee2f339aa5559570266c23fdd39f6fb3d7
Oct 30 21:15:04.352598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.352567    4918 generic.go:356] "Generic (PLEG): container finished" podID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" containerID="3008b33f705783db731ea00ba92998ece4666aca6837411f19f81821749aba7e" exitCode=60
Oct 30 21:15:04.353034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.352665    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5" event={"ID":"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf","Type":"ContainerDied","Data":"3008b33f705783db731ea00ba92998ece4666aca6837411f19f81821749aba7e"}
Oct 30 21:15:04.353034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.352717    4918 scope.go:122] "RemoveContainer" containerID="9ad915f5fc7306b37a176a20ba027474194ad04292060b2ea425919dacb74fa8"
Oct 30 21:15:04.353034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.352899    4918 scope.go:122] "RemoveContainer" containerID="3008b33f705783db731ea00ba92998ece4666aca6837411f19f81821749aba7e"
Oct 30 21:15:04.353231 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:04.353212    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-7l7f5_odh-ai-gateway-infra(2fb9b6b2-bcbf-4439-b88c-32ccba5878cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf"
Oct 30 21:15:04.354633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.354612    4918 generic.go:356] "Generic (PLEG): container finished" podID="3615fef6-3ddb-47ba-b359-a8431b57d513" containerID="4b59c0d57078335f9c5b6cc1ef21edee88ff43a18979af25f90ccbd984d6e417" exitCode=60
Oct 30 21:15:04.354765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.354700    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt" event={"ID":"3615fef6-3ddb-47ba-b359-a8431b57d513","Type":"ContainerDied","Data":"4b59c0d57078335f9c5b6cc1ef21edee88ff43a18979af25f90ccbd984d6e417"}
Oct 30 21:15:04.354914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.354896    4918 scope.go:122] "RemoveContainer" containerID="4b59c0d57078335f9c5b6cc1ef21edee88ff43a18979af25f90ccbd984d6e417"
Oct 30 21:15:04.355208 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:04.355186    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt_odh-ai-gateway-infra(3615fef6-3ddb-47ba-b359-a8431b57d513)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513"
Oct 30 21:15:04.356378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.356353    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx" event={"ID":"9711c192-fc19-4c13-8615-1b3d3bb945df","Type":"ContainerStarted","Data":"ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873"}
Oct 30 21:15:04.356506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.356417    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:04.356506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.356437    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx" event={"ID":"9711c192-fc19-4c13-8615-1b3d3bb945df","Type":"ContainerStarted","Data":"dcad2ab290f77cbc0ecc7964246ce012944b6989225c363e7fdd50910cc3757c"}
Oct 30 21:15:04.358208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.358191    4918 scope.go:122] "RemoveContainer" containerID="8eea5367f407e9ecb96e8ea301ba13a218124be7f911ed163dbbbf1b1a7fd309"
Oct 30 21:15:04.358493 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:04.358470    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p_odh-ai-gateway-infra(71b6f57f-52a0-460c-87ae-db559858428b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p" podUID="71b6f57f-52a0-460c-87ae-db559858428b"
Oct 30 21:15:04.359216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.359197    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm" event={"ID":"d9dbe6fd-1277-4644-9906-0fb0797bb199","Type":"ContainerStarted","Data":"c8436c929a6e76a2d21a71694b2a31ee2f339aa5559570266c23fdd39f6fb3d7"}
Oct 30 21:15:04.360865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.360849    4918 scope.go:122] "RemoveContainer" containerID="e6574ddd543984063877aa3961dedd10b1d80feb8a1ff130158c8458c3ca0d72"
Oct 30 21:15:04.361071 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:04.361054    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d_odh-ai-gateway-infra(681c66ea-6c52-45ac-a2b5-6e4ea80188be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be"
Oct 30 21:15:04.390224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.390180    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx" podStartSLOduration=2.390168382 podStartE2EDuration="2.390168382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:04.388600638 +0000 UTC m=+817.435159804" watchObservedRunningTime="2026-10-30 21:15:04.390168382 +0000 UTC m=+817.436727527"
Oct 30 21:15:04.391834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.391813    4918 scope.go:122] "RemoveContainer" containerID="2727cb692ece0f42afac8097b9be398330056e720595397ce74e15a1331730eb"
Oct 30 21:15:04.604191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.604159    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"]
Oct 30 21:15:04.608218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.608195    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.622222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.622198    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"]
Oct 30 21:15:04.728921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.728878    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.729115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.728984    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4x4c\" (UniqueName: \"kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-kube-api-access-t4x4c\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.729115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.729022    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae65e4e9-386b-440c-b568-2402f61346b4-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.729115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.729050    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae65e4e9-386b-440c-b568-2402f61346b4-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.729115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.729092    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.729317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.729116    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.729317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.729150    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.729317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.729223    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.729317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.729261    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.829952    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.830040    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4x4c\" (UniqueName: \"kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-kube-api-access-t4x4c\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.830077    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae65e4e9-386b-440c-b568-2402f61346b4-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.830107    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae65e4e9-386b-440c-b568-2402f61346b4-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.830145    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.830169    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.830202    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.830274    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.830325    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.830576    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.830657    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.830759    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.830883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.830868    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.831276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.831255    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae65e4e9-386b-440c-b568-2402f61346b4-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.832630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.832608    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.832883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.832866    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae65e4e9-386b-440c-b568-2402f61346b4-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.838879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.838848    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.838953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.838876    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4x4c\" (UniqueName: \"kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-kube-api-access-t4x4c\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:04.929577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:04.929537    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:05.076555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.076528    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b1aadb-6171-4482-b5bf-9c9773b18f3c" path="/var/lib/kubelet/pods/76b1aadb-6171-4482-b5bf-9c9773b18f3c/volumes"
Oct 30 21:15:05.212627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.212593    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"]
Oct 30 21:15:05.213492 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:05.213462    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f WatchSource:0}: Error finding container 2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f: Status 404 returned error can't find the container with id 2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f
Oct 30 21:15:05.216145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.216107    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:05.216228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.216196    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:05.216270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.216240    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:05.376284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.376247    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" event={"ID":"ae65e4e9-386b-440c-b568-2402f61346b4","Type":"ContainerStarted","Data":"a5c5c7f68ff7db45cf1f20fe8a2fa57dcc30f9d3df4b5d07b0121ac0ae418883"}
Oct 30 21:15:05.376802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.376310    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" event={"ID":"ae65e4e9-386b-440c-b568-2402f61346b4","Type":"ContainerStarted","Data":"2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f"}
Oct 30 21:15:05.378569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.378546    4918 scope.go:122] "RemoveContainer" containerID="3008b33f705783db731ea00ba92998ece4666aca6837411f19f81821749aba7e"
Oct 30 21:15:05.378866 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:05.378841    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-7l7f5_odh-ai-gateway-infra(2fb9b6b2-bcbf-4439-b88c-32ccba5878cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf"
Oct 30 21:15:05.380406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.380374    4918 scope.go:122] "RemoveContainer" containerID="4b59c0d57078335f9c5b6cc1ef21edee88ff43a18979af25f90ccbd984d6e417"
Oct 30 21:15:05.380647 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:05.380626    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt_odh-ai-gateway-infra(3615fef6-3ddb-47ba-b359-a8431b57d513)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513"
Oct 30 21:15:05.390371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.390298    4918 generic.go:356] "Generic (PLEG): container finished" podID="d9dbe6fd-1277-4644-9906-0fb0797bb199" containerID="dba9860a377cd7c3cb771be62a6db4bfb6241a75678890c55aac4d49289b778a" exitCode=0
Oct 30 21:15:05.391317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.391286    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm" event={"ID":"d9dbe6fd-1277-4644-9906-0fb0797bb199","Type":"ContainerDied","Data":"dba9860a377cd7c3cb771be62a6db4bfb6241a75678890c55aac4d49289b778a"}
Oct 30 21:15:05.418780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.418730    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" podStartSLOduration=1.418714745 podStartE2EDuration="1.418714745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:05.416059734 +0000 UTC m=+818.462618918" watchObservedRunningTime="2026-10-30 21:15:05.418714745 +0000 UTC m=+818.465273891"
Oct 30 21:15:05.930651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:05.930598    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:06.584802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:06.584777    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:06.661382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:06.661312    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-openshift-ca\") pod \"d9dbe6fd-1277-4644-9906-0fb0797bb199\" (UID: \"d9dbe6fd-1277-4644-9906-0fb0797bb199\") "
Oct 30 21:15:06.661382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:06.661368    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9dbe6fd-1277-4644-9906-0fb0797bb199-kube-api-access-cpwmp\" (UniqueName: \"kubernetes.io/projected/d9dbe6fd-1277-4644-9906-0fb0797bb199-kube-api-access-cpwmp\") pod \"d9dbe6fd-1277-4644-9906-0fb0797bb199\" (UID: \"d9dbe6fd-1277-4644-9906-0fb0797bb199\") "
Oct 30 21:15:06.661613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:06.661433    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-xks-ca\") pod \"d9dbe6fd-1277-4644-9906-0fb0797bb199\" (UID: \"d9dbe6fd-1277-4644-9906-0fb0797bb199\") "
Oct 30 21:15:06.661795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:06.661768    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-openshift-ca" pod "d9dbe6fd-1277-4644-9906-0fb0797bb199" (UID: "d9dbe6fd-1277-4644-9906-0fb0797bb199"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:06.663901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:06.663869    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9dbe6fd-1277-4644-9906-0fb0797bb199-kube-api-access-cpwmp" pod "d9dbe6fd-1277-4644-9906-0fb0797bb199" (UID: "d9dbe6fd-1277-4644-9906-0fb0797bb199"). InnerVolumeSpecName "kube-api-access-cpwmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:06.664067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:06.664047    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-xks-ca" pod "d9dbe6fd-1277-4644-9906-0fb0797bb199" (UID: "d9dbe6fd-1277-4644-9906-0fb0797bb199"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:06.763010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:06.762977    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:06.763010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:06.763006    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpwmp\" (UniqueName: \"kubernetes.io/projected/d9dbe6fd-1277-4644-9906-0fb0797bb199-kube-api-access-cpwmp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:06.763010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:06.763018    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9dbe6fd-1277-4644-9906-0fb0797bb199-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:06.931660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:06.931584    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:15:06.931660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:06.931638    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" podUID="ae65e4e9-386b-440c-b568-2402f61346b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 21:15:07.420901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:07.420870    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"
Oct 30 21:15:07.421058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:07.420897    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm" event={"ID":"d9dbe6fd-1277-4644-9906-0fb0797bb199","Type":"ContainerDied","Data":"c8436c929a6e76a2d21a71694b2a31ee2f339aa5559570266c23fdd39f6fb3d7"}
Oct 30 21:15:07.421058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:07.420954    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8436c929a6e76a2d21a71694b2a31ee2f339aa5559570266c23fdd39f6fb3d7"
Oct 30 21:15:07.934421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:07.931806    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:15:07.934421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:07.931863    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" podUID="ae65e4e9-386b-440c-b568-2402f61346b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 21:15:08.934424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:08.933847    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:15:08.934424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:08.933905    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" podUID="ae65e4e9-386b-440c-b568-2402f61346b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 21:15:09.189472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:09.188916    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:15:09.934415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:09.933864    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:15:09.934415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:09.933941    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" podUID="ae65e4e9-386b-440c-b568-2402f61346b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 21:15:10.152414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:10.152023    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:15:10.244422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:10.226797    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:15:10.931007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:10.930968    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:15:10.931185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:10.931053    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" podUID="ae65e4e9-386b-440c-b568-2402f61346b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 21:15:11.930424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:11.930375    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:15:11.930817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:11.930457    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" podUID="ae65e4e9-386b-440c-b568-2402f61346b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 21:15:13.848039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:13.848008    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:13.848527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:13.848490    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:13.849789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:13.849771    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:15:14.070561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:14.070533    4918 scope.go:122] "RemoveContainer" containerID="e6574ddd543984063877aa3961dedd10b1d80feb8a1ff130158c8458c3ca0d72"
Oct 30 21:15:14.587657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:14.587628    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d" event={"ID":"681c66ea-6c52-45ac-a2b5-6e4ea80188be","Type":"ContainerStarted","Data":"7424e4b5771be24aa10ce1aedf31d1d30bae6b4ea43577764698a69c90296ebb"}
Oct 30 21:15:14.618188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:14.618108    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d" podStartSLOduration=14.618067805 podStartE2EDuration="14.618067805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:14.609850647 +0000 UTC m=+827.656409813" watchObservedRunningTime="2026-10-30 21:15:14.618067805 +0000 UTC m=+827.664626952"
Oct 30 21:15:15.079813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:15.079780    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-698f9c649-d76kx"]
Oct 30 21:15:15.083042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:15.083021    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:15.595764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:15.595732    4918 generic.go:356] "Generic (PLEG): container finished" podID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" containerID="7424e4b5771be24aa10ce1aedf31d1d30bae6b4ea43577764698a69c90296ebb" exitCode=60
Oct 30 21:15:15.595919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:15.595802    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d" event={"ID":"681c66ea-6c52-45ac-a2b5-6e4ea80188be","Type":"ContainerDied","Data":"7424e4b5771be24aa10ce1aedf31d1d30bae6b4ea43577764698a69c90296ebb"}
Oct 30 21:15:15.595919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:15.595872    4918 scope.go:122] "RemoveContainer" containerID="e6574ddd543984063877aa3961dedd10b1d80feb8a1ff130158c8458c3ca0d72"
Oct 30 21:15:15.596283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:15.596260    4918 scope.go:122] "RemoveContainer" containerID="7424e4b5771be24aa10ce1aedf31d1d30bae6b4ea43577764698a69c90296ebb"
Oct 30 21:15:15.596556 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:15.596538    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d_odh-ai-gateway-infra(681c66ea-6c52-45ac-a2b5-6e4ea80188be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be"
Oct 30 21:15:15.652879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:15.652849    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"]
Oct 30 21:15:16.069685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.069659    4918 scope.go:122] "RemoveContainer" containerID="3008b33f705783db731ea00ba92998ece4666aca6837411f19f81821749aba7e"
Oct 30 21:15:16.161515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.161483    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"]
Oct 30 21:15:16.162256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.162240    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d9dbe6fd-1277-4644-9906-0fb0797bb199" containerName="revoke-keys"
Oct 30 21:15:16.162362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.162258    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9dbe6fd-1277-4644-9906-0fb0797bb199" containerName="revoke-keys"
Oct 30 21:15:16.162457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.162419    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9dbe6fd-1277-4644-9906-0fb0797bb199" containerName="revoke-keys"
Oct 30 21:15:16.167520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.167499    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:16.181251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.181228    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:16.182940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.182921    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"]
Oct 30 21:15:16.263673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.263644    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27d0b599-v7c59\" (UID: \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:16.263840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.263696    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27d0b599-v7c59\" (UID: \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:16.263840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.263822    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvvj\" (UniqueName: \"kubernetes.io/projected/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-kube-api-access-6bvvj\") pod \"maas-api-revoke-sub-27d0b599-v7c59\" (UID: \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:16.364983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.364908    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvvj\" (UniqueName: \"kubernetes.io/projected/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-kube-api-access-6bvvj\") pod \"maas-api-revoke-sub-27d0b599-v7c59\" (UID: \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:16.365119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.365017    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27d0b599-v7c59\" (UID: \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:16.365119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.365057    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27d0b599-v7c59\" (UID: \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:16.365736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.365693    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27d0b599-v7c59\" (UID: \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:16.367648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.367630    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27d0b599-v7c59\" (UID: \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:16.385356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.385331    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvvj\" (UniqueName: \"kubernetes.io/projected/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-kube-api-access-6bvvj\") pod \"maas-api-revoke-sub-27d0b599-v7c59\" (UID: \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:16.480971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.480925    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:16.605845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.605799    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5" event={"ID":"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf","Type":"ContainerStarted","Data":"13dc07bde39466a6c4442d2f3298f980c8ee5cfb2f385256d5c1fb95db56c932"}
Oct 30 21:15:16.740937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.740909    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"
Oct 30 21:15:16.774951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.774922    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"]
Oct 30 21:15:16.777827 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:16.777113    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d9481e_d22b_4e57_b2d3_d8edfdb3a1ff.slice/crio-f1034b16e623a77ae631ae4d741cd1563404f5921b0c5a11025cfdbb276bba8f WatchSource:0}: Error finding container f1034b16e623a77ae631ae4d741cd1563404f5921b0c5a11025cfdbb276bba8f: Status 404 returned error can't find the container with id f1034b16e623a77ae631ae4d741cd1563404f5921b0c5a11025cfdbb276bba8f
Oct 30 21:15:16.877111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.877088    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681c66ea-6c52-45ac-a2b5-6e4ea80188be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681c66ea-6c52-45ac-a2b5-6e4ea80188be-maas-api-service-ca\") pod \"681c66ea-6c52-45ac-a2b5-6e4ea80188be\" (UID: \"681c66ea-6c52-45ac-a2b5-6e4ea80188be\") "
Oct 30 21:15:16.877255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.877140    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681c66ea-6c52-45ac-a2b5-6e4ea80188be-kube-api-access-9bqf7\" (UniqueName: \"kubernetes.io/projected/681c66ea-6c52-45ac-a2b5-6e4ea80188be-kube-api-access-9bqf7\") pod \"681c66ea-6c52-45ac-a2b5-6e4ea80188be\" (UID: \"681c66ea-6c52-45ac-a2b5-6e4ea80188be\") "
Oct 30 21:15:16.877522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.877498    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681c66ea-6c52-45ac-a2b5-6e4ea80188be-maas-api-service-ca" pod "681c66ea-6c52-45ac-a2b5-6e4ea80188be" (UID: "681c66ea-6c52-45ac-a2b5-6e4ea80188be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:16.879220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.879201    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681c66ea-6c52-45ac-a2b5-6e4ea80188be-kube-api-access-9bqf7" pod "681c66ea-6c52-45ac-a2b5-6e4ea80188be" (UID: "681c66ea-6c52-45ac-a2b5-6e4ea80188be"). InnerVolumeSpecName "kube-api-access-9bqf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:16.978981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.978953    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/681c66ea-6c52-45ac-a2b5-6e4ea80188be-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:16.978981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:16.978977    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bqf7\" (UniqueName: \"kubernetes.io/projected/681c66ea-6c52-45ac-a2b5-6e4ea80188be-kube-api-access-9bqf7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:17.617463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.617435    4918 generic.go:356] "Generic (PLEG): container finished" podID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" containerID="13dc07bde39466a6c4442d2f3298f980c8ee5cfb2f385256d5c1fb95db56c932" exitCode=60
Oct 30 21:15:17.617840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.617512    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5" event={"ID":"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf","Type":"ContainerDied","Data":"13dc07bde39466a6c4442d2f3298f980c8ee5cfb2f385256d5c1fb95db56c932"}
Oct 30 21:15:17.617840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.617618    4918 scope.go:122] "RemoveContainer" containerID="3008b33f705783db731ea00ba92998ece4666aca6837411f19f81821749aba7e"
Oct 30 21:15:17.618004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.617987    4918 scope.go:122] "RemoveContainer" containerID="13dc07bde39466a6c4442d2f3298f980c8ee5cfb2f385256d5c1fb95db56c932"
Oct 30 21:15:17.618352 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:17.618325    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889915-7l7f5_odh-ai-gateway-infra(2fb9b6b2-bcbf-4439-b88c-32ccba5878cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf"
Oct 30 21:15:17.619138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.619122    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"
Oct 30 21:15:17.619221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.619164    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d" event={"ID":"681c66ea-6c52-45ac-a2b5-6e4ea80188be","Type":"ContainerDied","Data":"2681d916b33b520226ad69752ca5431851c69b546ba0063d9f8db6b59eca8183"}
Oct 30 21:15:17.620763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.620740    4918 generic.go:356] "Generic (PLEG): container finished" podID="09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff" containerID="8851ae03b618000b24d6082a71096a74615f2ca72f54d47f3e8248c76a3a1b38" exitCode=0
Oct 30 21:15:17.620847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.620780    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59" event={"ID":"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff","Type":"ContainerDied","Data":"8851ae03b618000b24d6082a71096a74615f2ca72f54d47f3e8248c76a3a1b38"}
Oct 30 21:15:17.620847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.620823    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59" event={"ID":"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff","Type":"ContainerStarted","Data":"f1034b16e623a77ae631ae4d741cd1563404f5921b0c5a11025cfdbb276bba8f"}
Oct 30 21:15:17.675742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.675709    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"]
Oct 30 21:15:17.677492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.677469    4918 scope.go:122] "RemoveContainer" containerID="7424e4b5771be24aa10ce1aedf31d1d30bae6b4ea43577764698a69c90296ebb"
Oct 30 21:15:17.724939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.724907    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"]
Oct 30 21:15:17.729695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:17.729659    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889915-q848d"]
Oct 30 21:15:18.069980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.069948    4918 scope.go:122] "RemoveContainer" containerID="4b59c0d57078335f9c5b6cc1ef21edee88ff43a18979af25f90ccbd984d6e417"
Oct 30 21:15:18.070174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.070040    4918 scope.go:122] "RemoveContainer" containerID="8eea5367f407e9ecb96e8ea301ba13a218124be7f911ed163dbbbf1b1a7fd309"
Oct 30 21:15:18.629559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.629521    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p" event={"ID":"71b6f57f-52a0-460c-87ae-db559858428b","Type":"ContainerStarted","Data":"7b899ea71ed09e248b653bb86c3a5fe05805879a3b7ae171b972c11b8f41de95"}
Oct 30 21:15:18.634400 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.634356    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt" event={"ID":"3615fef6-3ddb-47ba-b359-a8431b57d513","Type":"ContainerStarted","Data":"737bbe88f660c30dc7d107cde6ed59388917c7943f9892bd66c7d42a596dbdf7"}
Oct 30 21:15:18.654772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.654724    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p" podStartSLOduration=18.654709876 podStartE2EDuration="18.654709876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:18.652647145 +0000 UTC m=+831.699206290" watchObservedRunningTime="2026-10-30 21:15:18.654709876 +0000 UTC m=+831.701269020"
Oct 30 21:15:18.803161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.803134    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"
Oct 30 21:15:18.817208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.817187    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:18.848214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.848173    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-maas-api-service-ca\") pod \"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf\" (UID: \"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf\") "
Oct 30 21:15:18.848433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.848403    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-kube-api-access-xghzw\" (UniqueName: \"kubernetes.io/projected/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-kube-api-access-xghzw\") pod \"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf\" (UID: \"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf\") "
Oct 30 21:15:18.849215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.849186    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-maas-api-service-ca" pod "2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" (UID: "2fb9b6b2-bcbf-4439-b88c-32ccba5878cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:18.851223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.851186    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-kube-api-access-xghzw" pod "2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" (UID: "2fb9b6b2-bcbf-4439-b88c-32ccba5878cf"). InnerVolumeSpecName "kube-api-access-xghzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:18.949569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.949535    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-openshift-ca\") pod \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\" (UID: \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\") "
Oct 30 21:15:18.949708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.949579    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-kube-api-access-6bvvj\" (UniqueName: \"kubernetes.io/projected/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-kube-api-access-6bvvj\") pod \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\" (UID: \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\") "
Oct 30 21:15:18.949708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.949640    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-xks-ca\") pod \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\" (UID: \"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff\") "
Oct 30 21:15:18.949940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.949915    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-openshift-ca" pod "09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff" (UID: "09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:18.950054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.950035    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xghzw\" (UniqueName: \"kubernetes.io/projected/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-kube-api-access-xghzw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:18.950054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.950052    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:18.950210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.950061    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:18.951733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.951713    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-xks-ca" pod "09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff" (UID: "09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:18.951791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:18.951754    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-kube-api-access-6bvvj" pod "09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff" (UID: "09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff"). InnerVolumeSpecName "kube-api-access-6bvvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:19.051702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.051623    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvvj\" (UniqueName: \"kubernetes.io/projected/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-kube-api-access-6bvvj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.051702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.051650    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:19.081069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.081043    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" path="/var/lib/kubelet/pods/681c66ea-6c52-45ac-a2b5-6e4ea80188be/volumes"
Oct 30 21:15:19.139277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139248    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"]
Oct 30 21:15:19.139765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139753    4918 cpu_manager.go:414] "containerMap: removing container" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" containerName="cleanup"
Oct 30 21:15:19.139807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139767    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" containerName="cleanup"
Oct 30 21:15:19.139807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139785    4918 cpu_manager.go:414] "containerMap: removing container" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" containerName="cleanup"
Oct 30 21:15:19.139807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139790    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" containerName="cleanup"
Oct 30 21:15:19.139807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139800    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" containerName="cleanup"
Oct 30 21:15:19.139807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139805    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" containerName="cleanup"
Oct 30 21:15:19.139967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139839    4918 cpu_manager.go:414] "containerMap: removing container" podUID="09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff" containerName="revoke-keys"
Oct 30 21:15:19.139967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139844    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff" containerName="revoke-keys"
Oct 30 21:15:19.139967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139853    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" containerName="cleanup"
Oct 30 21:15:19.139967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139857    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" containerName="cleanup"
Oct 30 21:15:19.139967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139944    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff" containerName="revoke-keys"
Oct 30 21:15:19.139967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139952    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" containerName="cleanup"
Oct 30 21:15:19.139967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139960    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" containerName="cleanup"
Oct 30 21:15:19.139967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139965    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" containerName="cleanup"
Oct 30 21:15:19.140192 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.139972    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" containerName="cleanup"
Oct 30 21:15:19.143250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.143234    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.153131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.153112    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-384164\"/\"distinct-w0-384164-kserve-self-signed-certs\""
Oct 30 21:15:19.158671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.158647    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"]
Oct 30 21:15:19.253491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.253459    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-tmp-dir\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.253614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.253564    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-model-cache\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.253614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.253605    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnhds\" (UniqueName: \"kubernetes.io/projected/e7a0864e-46fb-42d8-892e-0579659d468d-kube-api-access-wnhds\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.253790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.253773    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-dshm\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.253846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.253827    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a0864e-46fb-42d8-892e-0579659d468d-tls-certs\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.253905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.253890    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-home\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.358495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.358144    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-model-cache\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.358495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.358224    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnhds\" (UniqueName: \"kubernetes.io/projected/e7a0864e-46fb-42d8-892e-0579659d468d-kube-api-access-wnhds\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.358495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.358404    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-dshm\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.358495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.358452    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a0864e-46fb-42d8-892e-0579659d468d-tls-certs\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.358495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.358495    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-home\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.358873 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.358602    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-tmp-dir\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.359124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.359078    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-tmp-dir\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.359443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.359383    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-model-cache\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.360778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.360752    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-home\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.367962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.365032    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a0864e-46fb-42d8-892e-0579659d468d-tls-certs\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.367962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.365420    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-dshm\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.385405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.385356    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnhds\" (UniqueName: \"kubernetes.io/projected/e7a0864e-46fb-42d8-892e-0579659d468d-kube-api-access-wnhds\") pod \"distinct-w0-384164-kserve-788d878db7-k9knv\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") " pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.513927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.513899    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:19.644686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.644585    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"
Oct 30 21:15:19.645101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.644587    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5" event={"ID":"2fb9b6b2-bcbf-4439-b88c-32ccba5878cf","Type":"ContainerDied","Data":"77c1aa475506a62f95d017c506d05de9ede2a829e08a37e20a4c74ba5fb4615a"}
Oct 30 21:15:19.645101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.644740    4918 scope.go:122] "RemoveContainer" containerID="13dc07bde39466a6c4442d2f3298f980c8ee5cfb2f385256d5c1fb95db56c932"
Oct 30 21:15:19.647613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.647585    4918 generic.go:356] "Generic (PLEG): container finished" podID="3615fef6-3ddb-47ba-b359-a8431b57d513" containerID="737bbe88f660c30dc7d107cde6ed59388917c7943f9892bd66c7d42a596dbdf7" exitCode=60
Oct 30 21:15:19.647761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.647744    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt" event={"ID":"3615fef6-3ddb-47ba-b359-a8431b57d513","Type":"ContainerDied","Data":"737bbe88f660c30dc7d107cde6ed59388917c7943f9892bd66c7d42a596dbdf7"}
Oct 30 21:15:19.648153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.648137    4918 scope.go:122] "RemoveContainer" containerID="737bbe88f660c30dc7d107cde6ed59388917c7943f9892bd66c7d42a596dbdf7"
Oct 30 21:15:19.648452 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:19.648425    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt_odh-ai-gateway-infra(3615fef6-3ddb-47ba-b359-a8431b57d513)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513"
Oct 30 21:15:19.650663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.650644    4918 generic.go:356] "Generic (PLEG): container finished" podID="71b6f57f-52a0-460c-87ae-db559858428b" containerID="7b899ea71ed09e248b653bb86c3a5fe05805879a3b7ae171b972c11b8f41de95" exitCode=60
Oct 30 21:15:19.650762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.650719    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p" event={"ID":"71b6f57f-52a0-460c-87ae-db559858428b","Type":"ContainerDied","Data":"7b899ea71ed09e248b653bb86c3a5fe05805879a3b7ae171b972c11b8f41de95"}
Oct 30 21:15:19.651073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.651047    4918 scope.go:122] "RemoveContainer" containerID="7b899ea71ed09e248b653bb86c3a5fe05805879a3b7ae171b972c11b8f41de95"
Oct 30 21:15:19.651333 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:19.651308    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p_odh-ai-gateway-infra(71b6f57f-52a0-460c-87ae-db559858428b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p" podUID="71b6f57f-52a0-460c-87ae-db559858428b"
Oct 30 21:15:19.652207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.652189    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59" event={"ID":"09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff","Type":"ContainerDied","Data":"f1034b16e623a77ae631ae4d741cd1563404f5921b0c5a11025cfdbb276bba8f"}
Oct 30 21:15:19.652289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.652222    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1034b16e623a77ae631ae4d741cd1563404f5921b0c5a11025cfdbb276bba8f"
Oct 30 21:15:19.652289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.652231    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"
Oct 30 21:15:19.655908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.655895    4918 scope.go:122] "RemoveContainer" containerID="4b59c0d57078335f9c5b6cc1ef21edee88ff43a18979af25f90ccbd984d6e417"
Oct 30 21:15:19.667681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.667661    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"]
Oct 30 21:15:19.669325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.669308    4918 scope.go:122] "RemoveContainer" containerID="8eea5367f407e9ecb96e8ea301ba13a218124be7f911ed163dbbbf1b1a7fd309"
Oct 30 21:15:19.673343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.673326    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"]
Oct 30 21:15:19.682031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.679269    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889915-7l7f5"]
Oct 30 21:15:19.746652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.746622    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"]
Oct 30 21:15:19.811351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:19.811323    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"]
Oct 30 21:15:19.812304 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:19.812276    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7a0864e_46fb_42d8_892e_0579659d468d.slice/crio-3b4afb33c04694814f4236e0e0f3fac53a502ede827b05af5ed335ac15c4ce58 WatchSource:0}: Error finding container 3b4afb33c04694814f4236e0e0f3fac53a502ede827b05af5ed335ac15c4ce58: Status 404 returned error can't find the container with id 3b4afb33c04694814f4236e0e0f3fac53a502ede827b05af5ed335ac15c4ce58
Oct 30 21:15:20.080862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.080826    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx" podUID="9711c192-fc19-4c13-8615-1b3d3bb945df" containerName="payload-processing" containerID="cri-o://ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873" gracePeriod=25
Oct 30 21:15:20.633984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.633954    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:20.758768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.755105    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-provider-credentials\") pod \"9711c192-fc19-4c13-8615-1b3d3bb945df\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") "
Oct 30 21:15:20.758768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.755287    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-kube-api-access-tbd84\" (UniqueName: \"kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-kube-api-access-tbd84\") pod \"9711c192-fc19-4c13-8615-1b3d3bb945df\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") "
Oct 30 21:15:20.758768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.755494    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-routing-overlay-volume\") pod \"9711c192-fc19-4c13-8615-1b3d3bb945df\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") "
Oct 30 21:15:20.758768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.755535    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-plugins-config-volume\") pod \"9711c192-fc19-4c13-8615-1b3d3bb945df\" (UID: \"9711c192-fc19-4c13-8615-1b3d3bb945df\") "
Oct 30 21:15:20.758768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.757654    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-plugins-config-volume" pod "9711c192-fc19-4c13-8615-1b3d3bb945df" (UID: "9711c192-fc19-4c13-8615-1b3d3bb945df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.758768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.758719    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-routing-overlay-volume" pod "9711c192-fc19-4c13-8615-1b3d3bb945df" (UID: "9711c192-fc19-4c13-8615-1b3d3bb945df"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:20.892416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.885275    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-kube-api-access-tbd84" pod "9711c192-fc19-4c13-8615-1b3d3bb945df" (UID: "9711c192-fc19-4c13-8615-1b3d3bb945df"). InnerVolumeSpecName "kube-api-access-tbd84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:20.892416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.885594    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-provider-credentials" pod "9711c192-fc19-4c13-8615-1b3d3bb945df" (UID: "9711c192-fc19-4c13-8615-1b3d3bb945df"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:20.917415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.899794    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" event={"ID":"e7a0864e-46fb-42d8-892e-0579659d468d","Type":"ContainerStarted","Data":"d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5"}
Oct 30 21:15:20.917415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.899853    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" event={"ID":"e7a0864e-46fb-42d8-892e-0579659d468d","Type":"ContainerStarted","Data":"3b4afb33c04694814f4236e0e0f3fac53a502ede827b05af5ed335ac15c4ce58"}
Oct 30 21:15:20.958122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.958067    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podStartSLOduration=1.958045491 podStartE2EDuration="1.958045491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:20.94717412 +0000 UTC m=+833.993733267" watchObservedRunningTime="2026-10-30 21:15:20.958045491 +0000 UTC m=+834.004604640"
Oct 30 21:15:20.983436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.980740    4918 generic.go:356] "Generic (PLEG): container finished" podID="9711c192-fc19-4c13-8615-1b3d3bb945df" containerID="ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873" exitCode=0
Oct 30 21:15:20.983436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.980922    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx" event={"ID":"9711c192-fc19-4c13-8615-1b3d3bb945df","Type":"ContainerDied","Data":"ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873"}
Oct 30 21:15:20.983436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.981013    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx" event={"ID":"9711c192-fc19-4c13-8615-1b3d3bb945df","Type":"ContainerDied","Data":"dcad2ab290f77cbc0ecc7964246ce012944b6989225c363e7fdd50910cc3757c"}
Oct 30 21:15:20.983436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.981068    4918 scope.go:122] "RemoveContainer" containerID="ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873"
Oct 30 21:15:20.983436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:20.981269    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698f9c649-d76kx"
Oct 30 21:15:21.030730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:21.030486    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:21.035419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:21.034452    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbd84\" (UniqueName: \"kubernetes.io/projected/9711c192-fc19-4c13-8615-1b3d3bb945df-kube-api-access-tbd84\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:21.035419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:21.034481    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:21.035419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:21.034507    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9711c192-fc19-4c13-8615-1b3d3bb945df-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:21.236417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:21.198582    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" path="/var/lib/kubelet/pods/2fb9b6b2-bcbf-4439-b88c-32ccba5878cf/volumes"
Oct 30 21:15:21.237206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:21.237177    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-698f9c649-d76kx"]
Oct 30 21:15:21.237370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:21.237356    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-698f9c649-d76kx"]
Oct 30 21:15:21.515309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:21.515277    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:21.644741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:21.622848    4918 scope.go:122] "RemoveContainer" containerID="ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873"
Oct 30 21:15:21.679417 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:21.656610    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873\": container with ID starting with ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873 not found: ID does not exist" containerID="ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873"
Oct 30 21:15:21.679770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:21.679694    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873"} err="failed to get container status \"ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873\": rpc error: code = NotFound desc = could not find container \"ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873\": container with ID starting with ada503e70322a153385f8c01f5a79ac49ca7f333a50d7593e55a7d69297c8873 not found: ID does not exist"
Oct 30 21:15:22.068433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.064549    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:15:22.219470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.214603    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"
Oct 30 21:15:22.219470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.217295    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:22.306616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.280974    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:22.306616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.284632    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p" event={"ID":"71b6f57f-52a0-460c-87ae-db559858428b","Type":"ContainerDied","Data":"f9e28941fa763fc92a900578032c33246042c5ab1c2b93749b7ca5059d58b855"}
Oct 30 21:15:22.306616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.284705    4918 scope.go:122] "RemoveContainer" containerID="7b899ea71ed09e248b653bb86c3a5fe05805879a3b7ae171b972c11b8f41de95"
Oct 30 21:15:22.384825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.382720    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b6f57f-52a0-460c-87ae-db559858428b-kube-api-access-5ntw7\" (UniqueName: \"kubernetes.io/projected/71b6f57f-52a0-460c-87ae-db559858428b-kube-api-access-5ntw7\") pod \"71b6f57f-52a0-460c-87ae-db559858428b\" (UID: \"71b6f57f-52a0-460c-87ae-db559858428b\") "
Oct 30 21:15:22.384825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.382831    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b6f57f-52a0-460c-87ae-db559858428b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b6f57f-52a0-460c-87ae-db559858428b-maas-api-service-ca\") pod \"71b6f57f-52a0-460c-87ae-db559858428b\" (UID: \"71b6f57f-52a0-460c-87ae-db559858428b\") "
Oct 30 21:15:22.421433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.411657    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b6f57f-52a0-460c-87ae-db559858428b-maas-api-service-ca" pod "71b6f57f-52a0-460c-87ae-db559858428b" (UID: "71b6f57f-52a0-460c-87ae-db559858428b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:22.487489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.484861    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b6f57f-52a0-460c-87ae-db559858428b-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.487489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.484929    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:15:22.491417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.487946    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:22.516418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.496039    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"
Oct 30 21:15:22.516418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.497020    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b6f57f-52a0-460c-87ae-db559858428b-kube-api-access-5ntw7" pod "71b6f57f-52a0-460c-87ae-db559858428b" (UID: "71b6f57f-52a0-460c-87ae-db559858428b"). InnerVolumeSpecName "kube-api-access-5ntw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:22.516418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.507847    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:22.526417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.517119    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:22.587424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.573554    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:22.620418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.601797    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3615fef6-3ddb-47ba-b359-a8431b57d513-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3615fef6-3ddb-47ba-b359-a8431b57d513-maas-api-service-ca\") pod \"3615fef6-3ddb-47ba-b359-a8431b57d513\" (UID: \"3615fef6-3ddb-47ba-b359-a8431b57d513\") "
Oct 30 21:15:22.620418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.601888    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3615fef6-3ddb-47ba-b359-a8431b57d513-kube-api-access-8jpzz\" (UniqueName: \"kubernetes.io/projected/3615fef6-3ddb-47ba-b359-a8431b57d513-kube-api-access-8jpzz\") pod \"3615fef6-3ddb-47ba-b359-a8431b57d513\" (UID: \"3615fef6-3ddb-47ba-b359-a8431b57d513\") "
Oct 30 21:15:22.620418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.602493    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ntw7\" (UniqueName: \"kubernetes.io/projected/71b6f57f-52a0-460c-87ae-db559858428b-kube-api-access-5ntw7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.620418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.607508    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3615fef6-3ddb-47ba-b359-a8431b57d513-maas-api-service-ca" pod "3615fef6-3ddb-47ba-b359-a8431b57d513" (UID: "3615fef6-3ddb-47ba-b359-a8431b57d513"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:22.725514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.724195    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3615fef6-3ddb-47ba-b359-a8431b57d513-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:22.725514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.724444    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3615fef6-3ddb-47ba-b359-a8431b57d513-kube-api-access-8jpzz" pod "3615fef6-3ddb-47ba-b359-a8431b57d513" (UID: "3615fef6-3ddb-47ba-b359-a8431b57d513"). InnerVolumeSpecName "kube-api-access-8jpzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:22.827416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:22.826536    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jpzz\" (UniqueName: \"kubernetes.io/projected/3615fef6-3ddb-47ba-b359-a8431b57d513-kube-api-access-8jpzz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:23.085469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.084954    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9711c192-fc19-4c13-8615-1b3d3bb945df" path="/var/lib/kubelet/pods/9711c192-fc19-4c13-8615-1b3d3bb945df/volumes"
Oct 30 21:15:23.337091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.335631    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt" event={"ID":"3615fef6-3ddb-47ba-b359-a8431b57d513","Type":"ContainerDied","Data":"1a863577f13a595748caa035153a6f7a91295628b13653ec02fadae60cfa1553"}
Oct 30 21:15:23.337091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.335701    4918 scope.go:122] "RemoveContainer" containerID="737bbe88f660c30dc7d107cde6ed59388917c7943f9892bd66c7d42a596dbdf7"
Oct 30 21:15:23.337091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.335838    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"
Oct 30 21:15:23.370759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.370729    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"]
Oct 30 21:15:23.386548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.379893    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889915-qjqnt"]
Oct 30 21:15:23.386548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.380838    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"
Oct 30 21:15:23.411027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.410992    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"]
Oct 30 21:15:23.419779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.419750    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889915-bmb8p"]
Oct 30 21:15:23.661613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.661521    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-w9kjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:15:23.661875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.661845    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp" podUID="de5f2249-cc22-496c-b84a-8d1fcf7c5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:23.710515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.710482    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-d54n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:15:23.710779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:23.710753    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7" podUID="40508368-7d69-41e3-a2ee-0be190a20281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:25.076243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.076214    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513" path="/var/lib/kubelet/pods/3615fef6-3ddb-47ba-b359-a8431b57d513/volumes"
Oct 30 21:15:25.076716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.076674    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b6f57f-52a0-460c-87ae-db559858428b" path="/var/lib/kubelet/pods/71b6f57f-52a0-460c-87ae-db559858428b/volumes"
Oct 30 21:15:25.141574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.141541    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"]
Oct 30 21:15:25.142288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142265    4918 cpu_manager.go:414] "containerMap: removing container" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513" containerName="cleanup"
Oct 30 21:15:25.142288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142286    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513" containerName="cleanup"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142300    4918 cpu_manager.go:414] "containerMap: removing container" podUID="71b6f57f-52a0-460c-87ae-db559858428b" containerName="cleanup"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142305    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b6f57f-52a0-460c-87ae-db559858428b" containerName="cleanup"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142311    4918 cpu_manager.go:414] "containerMap: removing container" podUID="71b6f57f-52a0-460c-87ae-db559858428b" containerName="cleanup"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142316    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b6f57f-52a0-460c-87ae-db559858428b" containerName="cleanup"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142326    4918 cpu_manager.go:414] "containerMap: removing container" podUID="71b6f57f-52a0-460c-87ae-db559858428b" containerName="cleanup"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142330    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b6f57f-52a0-460c-87ae-db559858428b" containerName="cleanup"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142342    4918 cpu_manager.go:414] "containerMap: removing container" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" containerName="cleanup"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142347    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" containerName="cleanup"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142354    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9711c192-fc19-4c13-8615-1b3d3bb945df" containerName="payload-processing"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142358    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9711c192-fc19-4c13-8615-1b3d3bb945df" containerName="payload-processing"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142365    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" containerName="cleanup"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142369    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" containerName="cleanup"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142404    4918 cpu_manager.go:414] "containerMap: removing container" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513" containerName="cleanup"
Oct 30 21:15:25.142449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142411    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513" containerName="cleanup"
Oct 30 21:15:25.142894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142485    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513" containerName="cleanup"
Oct 30 21:15:25.142894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142492    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="681c66ea-6c52-45ac-a2b5-6e4ea80188be" containerName="cleanup"
Oct 30 21:15:25.142894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142499    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513" containerName="cleanup"
Oct 30 21:15:25.142894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142504    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b6f57f-52a0-460c-87ae-db559858428b" containerName="cleanup"
Oct 30 21:15:25.142894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142510    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b6f57f-52a0-460c-87ae-db559858428b" containerName="cleanup"
Oct 30 21:15:25.142894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142517    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b6f57f-52a0-460c-87ae-db559858428b" containerName="cleanup"
Oct 30 21:15:25.142894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142524    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9711c192-fc19-4c13-8615-1b3d3bb945df" containerName="payload-processing"
Oct 30 21:15:25.142894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.142533    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb9b6b2-bcbf-4439-b88c-32ccba5878cf" containerName="cleanup"
Oct 30 21:15:25.147307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.147291    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:25.152799 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.152774    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:25.152949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.152848    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:15:25.152949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.152944    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:15:25.153555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.153485    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:15:25.157539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.157519    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"]
Oct 30 21:15:25.279005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.278972    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8334ee8-bprth\" (UID: \"0249779b-f7da-435d-ac3d-5c77766db493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:25.279180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.279117    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfwb7\" (UniqueName: \"kubernetes.io/projected/0249779b-f7da-435d-ac3d-5c77766db493-kube-api-access-vfwb7\") pod \"maas-api-revoke-sub-c8334ee8-bprth\" (UID: \"0249779b-f7da-435d-ac3d-5c77766db493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:25.279180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.279152    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8334ee8-bprth\" (UID: \"0249779b-f7da-435d-ac3d-5c77766db493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:25.380636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.380546    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8334ee8-bprth\" (UID: \"0249779b-f7da-435d-ac3d-5c77766db493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:25.380978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.380941    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfwb7\" (UniqueName: \"kubernetes.io/projected/0249779b-f7da-435d-ac3d-5c77766db493-kube-api-access-vfwb7\") pod \"maas-api-revoke-sub-c8334ee8-bprth\" (UID: \"0249779b-f7da-435d-ac3d-5c77766db493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:25.381077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.380996    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8334ee8-bprth\" (UID: \"0249779b-f7da-435d-ac3d-5c77766db493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:25.381916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.381894    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8334ee8-bprth\" (UID: \"0249779b-f7da-435d-ac3d-5c77766db493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:25.384753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.384732    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8334ee8-bprth\" (UID: \"0249779b-f7da-435d-ac3d-5c77766db493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:25.391565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.391538    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfwb7\" (UniqueName: \"kubernetes.io/projected/0249779b-f7da-435d-ac3d-5c77766db493-kube-api-access-vfwb7\") pod \"maas-api-revoke-sub-c8334ee8-bprth\" (UID: \"0249779b-f7da-435d-ac3d-5c77766db493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:25.462223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.462197    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:25.741896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:25.741863    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"]
Oct 30 21:15:25.743222 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:25.743188    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0249779b_f7da_435d_ac3d_5c77766db493.slice/crio-96f8e49f4edb9b7a6231cf96c1f5fc1a7699bbd2b92d89dfcca531fe874b50bb WatchSource:0}: Error finding container 96f8e49f4edb9b7a6231cf96c1f5fc1a7699bbd2b92d89dfcca531fe874b50bb: Status 404 returned error can't find the container with id 96f8e49f4edb9b7a6231cf96c1f5fc1a7699bbd2b92d89dfcca531fe874b50bb
Oct 30 21:15:26.437634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:26.437550    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth" event={"ID":"0249779b-f7da-435d-ac3d-5c77766db493","Type":"ContainerStarted","Data":"89ea5ea799292da486df529a8c0099d32604c86b5d1c83998c14cee6d4e76de0"}
Oct 30 21:15:26.437634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:26.437606    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth" event={"ID":"0249779b-f7da-435d-ac3d-5c77766db493","Type":"ContainerStarted","Data":"96f8e49f4edb9b7a6231cf96c1f5fc1a7699bbd2b92d89dfcca531fe874b50bb"}
Oct 30 21:15:26.454215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:26.454162    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth" podStartSLOduration=1.454145434 podStartE2EDuration="1.454145434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:26.453321774 +0000 UTC m=+839.499880918" watchObservedRunningTime="2026-10-30 21:15:26.454145434 +0000 UTC m=+839.500704648"
Oct 30 21:15:27.290742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.290709    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"]
Oct 30 21:15:27.291213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.291200    4918 cpu_manager.go:414] "containerMap: removing container" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513" containerName="cleanup"
Oct 30 21:15:27.291262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.291215    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513" containerName="cleanup"
Oct 30 21:15:27.291326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.291317    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3615fef6-3ddb-47ba-b359-a8431b57d513" containerName="cleanup"
Oct 30 21:15:27.296874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.296853    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:27.307529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.307491    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"]
Oct 30 21:15:27.407580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.407549    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4b43959-fhnn2\" (UID: \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:27.407776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.407662    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shxbs\" (UniqueName: \"kubernetes.io/projected/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-kube-api-access-shxbs\") pod \"maas-api-revoke-sub-d4b43959-fhnn2\" (UID: \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:27.407829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.407774    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4b43959-fhnn2\" (UID: \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:27.445498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.445470    4918 generic.go:356] "Generic (PLEG): container finished" podID="0249779b-f7da-435d-ac3d-5c77766db493" containerID="89ea5ea799292da486df529a8c0099d32604c86b5d1c83998c14cee6d4e76de0" exitCode=0
Oct 30 21:15:27.445862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.445527    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth" event={"ID":"0249779b-f7da-435d-ac3d-5c77766db493","Type":"ContainerDied","Data":"89ea5ea799292da486df529a8c0099d32604c86b5d1c83998c14cee6d4e76de0"}
Oct 30 21:15:27.508539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.508508    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4b43959-fhnn2\" (UID: \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:27.508734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.508659    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4b43959-fhnn2\" (UID: \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:27.508734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.508717    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shxbs\" (UniqueName: \"kubernetes.io/projected/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-kube-api-access-shxbs\") pod \"maas-api-revoke-sub-d4b43959-fhnn2\" (UID: \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:27.509144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.509125    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4b43959-fhnn2\" (UID: \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:27.511152 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.511130    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4b43959-fhnn2\" (UID: \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:27.518291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.518262    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shxbs\" (UniqueName: \"kubernetes.io/projected/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-kube-api-access-shxbs\") pod \"maas-api-revoke-sub-d4b43959-fhnn2\" (UID: \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:27.613130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.613095    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:27.886738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.886658    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"]
Oct 30 21:15:27.892327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.892306    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:27.904849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.904267    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"]
Oct 30 21:15:27.911669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.911627    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197109cc-f204-4d13-bf73-02172b3003ec-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:27.911784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.911751    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:27.911889 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.911847    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr9k8\" (UniqueName: \"kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-kube-api-access-sr9k8\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:27.912432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.911968    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:27.912432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.912101    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:27.912432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.912144    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:27.912432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.912194    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197109cc-f204-4d13-bf73-02172b3003ec-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:27.912432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.912231    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:27.912432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.912280    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:27.925448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:27.925360    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"]
Oct 30 21:15:28.012940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.012912    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.013088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.012967    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.013088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.013020    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197109cc-f204-4d13-bf73-02172b3003ec-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.013088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.013050    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.013088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.013081    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.013279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.013110    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197109cc-f204-4d13-bf73-02172b3003ec-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.013279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.013152    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.013279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.013222    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr9k8\" (UniqueName: \"kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-kube-api-access-sr9k8\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.013279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.013264    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.013670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.013651    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.013754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.013690    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.013754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.013718    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.014354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.014326    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197109cc-f204-4d13-bf73-02172b3003ec-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.014692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.014671    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.015745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.015723    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.015883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.015864    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197109cc-f204-4d13-bf73-02172b3003ec-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.021636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.021610    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.021723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.021611    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr9k8\" (UniqueName: \"kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-kube-api-access-sr9k8\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.230468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.230445    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:28.453220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.453122    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2" event={"ID":"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f","Type":"ContainerStarted","Data":"e0dbc1e5c2ea3378b127061d625d1e356c8dd4b83f9c3cc7f7da128e7754ebb2"}
Oct 30 21:15:28.453220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.453186    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2" event={"ID":"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f","Type":"ContainerStarted","Data":"793fc4c2c8955ec3fd54c13ebe9cc0b5f0f822c78c8c26ffc6ffd26be63d0916"}
Oct 30 21:15:28.475995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.475947    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2" podStartSLOduration=1.475933267 podStartE2EDuration="1.475933267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:28.471285138 +0000 UTC m=+841.517844284" watchObservedRunningTime="2026-10-30 21:15:28.475933267 +0000 UTC m=+841.522492449"
Oct 30 21:15:28.512821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.512786    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"]
Oct 30 21:15:28.522202 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:28.522172    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197109cc_f204_4d13_bf73_02172b3003ec.slice/crio-39e7f6ea793dfd2b980e487134be2d241494c33ed76e22b25fb9e86a075a484d WatchSource:0}: Error finding container 39e7f6ea793dfd2b980e487134be2d241494c33ed76e22b25fb9e86a075a484d: Status 404 returned error can't find the container with id 39e7f6ea793dfd2b980e487134be2d241494c33ed76e22b25fb9e86a075a484d
Oct 30 21:15:28.525751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.525713    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:28.525859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.525803    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:28.525905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.525854    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:28.610768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.610744    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:28.622319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.622294    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-openshift-ca\") pod \"0249779b-f7da-435d-ac3d-5c77766db493\" (UID: \"0249779b-f7da-435d-ac3d-5c77766db493\") "
Oct 30 21:15:28.622521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.622501    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0249779b-f7da-435d-ac3d-5c77766db493-kube-api-access-vfwb7\" (UniqueName: \"kubernetes.io/projected/0249779b-f7da-435d-ac3d-5c77766db493-kube-api-access-vfwb7\") pod \"0249779b-f7da-435d-ac3d-5c77766db493\" (UID: \"0249779b-f7da-435d-ac3d-5c77766db493\") "
Oct 30 21:15:28.622573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.622552    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-xks-ca\") pod \"0249779b-f7da-435d-ac3d-5c77766db493\" (UID: \"0249779b-f7da-435d-ac3d-5c77766db493\") "
Oct 30 21:15:28.622677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.622650    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-openshift-ca" pod "0249779b-f7da-435d-ac3d-5c77766db493" (UID: "0249779b-f7da-435d-ac3d-5c77766db493"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:28.623235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.623133    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:28.624678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.624654    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0249779b-f7da-435d-ac3d-5c77766db493-kube-api-access-vfwb7" pod "0249779b-f7da-435d-ac3d-5c77766db493" (UID: "0249779b-f7da-435d-ac3d-5c77766db493"). InnerVolumeSpecName "kube-api-access-vfwb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:28.625036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.625011    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-xks-ca" pod "0249779b-f7da-435d-ac3d-5c77766db493" (UID: "0249779b-f7da-435d-ac3d-5c77766db493"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:28.724757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.724658    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfwb7\" (UniqueName: \"kubernetes.io/projected/0249779b-f7da-435d-ac3d-5c77766db493-kube-api-access-vfwb7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:28.724757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.724690    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0249779b-f7da-435d-ac3d-5c77766db493-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:28.756650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.756611    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"]
Oct 30 21:15:28.757369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.757349    4918 cpu_manager.go:414] "containerMap: removing container" podUID="0249779b-f7da-435d-ac3d-5c77766db493" containerName="revoke-keys"
Oct 30 21:15:28.757480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.757374    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0249779b-f7da-435d-ac3d-5c77766db493" containerName="revoke-keys"
Oct 30 21:15:28.757597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.757583    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0249779b-f7da-435d-ac3d-5c77766db493" containerName="revoke-keys"
Oct 30 21:15:28.761453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.761431    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:28.772022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.771884    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"]
Oct 30 21:15:28.826194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.826165    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ff66b4f-kp68k\" (UID: \"2a85fae0-d8ef-4579-806b-54d744619594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:28.826381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.826362    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ff66b4f-kp68k\" (UID: \"2a85fae0-d8ef-4579-806b-54d744619594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:28.826524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.826501    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzqg\" (UniqueName: \"kubernetes.io/projected/2a85fae0-d8ef-4579-806b-54d744619594-kube-api-access-wjzqg\") pod \"maas-api-revoke-sub-5ff66b4f-kp68k\" (UID: \"2a85fae0-d8ef-4579-806b-54d744619594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:28.927721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.927683    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ff66b4f-kp68k\" (UID: \"2a85fae0-d8ef-4579-806b-54d744619594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:28.927908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.927745    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzqg\" (UniqueName: \"kubernetes.io/projected/2a85fae0-d8ef-4579-806b-54d744619594-kube-api-access-wjzqg\") pod \"maas-api-revoke-sub-5ff66b4f-kp68k\" (UID: \"2a85fae0-d8ef-4579-806b-54d744619594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:28.927908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.927817    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ff66b4f-kp68k\" (UID: \"2a85fae0-d8ef-4579-806b-54d744619594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:28.928320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.928299    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ff66b4f-kp68k\" (UID: \"2a85fae0-d8ef-4579-806b-54d744619594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:28.930303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.930281    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ff66b4f-kp68k\" (UID: \"2a85fae0-d8ef-4579-806b-54d744619594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:28.936367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:28.936344    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzqg\" (UniqueName: \"kubernetes.io/projected/2a85fae0-d8ef-4579-806b-54d744619594-kube-api-access-wjzqg\") pod \"maas-api-revoke-sub-5ff66b4f-kp68k\" (UID: \"2a85fae0-d8ef-4579-806b-54d744619594\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:29.106014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.086635    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:29.174901 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:29.172913    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:15:29.273440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.273382    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"]
Oct 30 21:15:29.279201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.279169    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.315858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.315818    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3011c6\"/\"distinct-2-w4-3011c6-kserve-self-signed-certs\""
Oct 30 21:15:29.320440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.319519    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"]
Oct 30 21:15:29.332052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.332021    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-home\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.332206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.332188    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-model-cache\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.332322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.332303    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-dshm\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.332421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.332370    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-tmp-dir\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.332490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.332468    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2f5787-d041-4361-aba9-f8c765ea3909-tls-certs\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.332577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.332561    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lz2l\" (UniqueName: \"kubernetes.io/projected/4a2f5787-d041-4361-aba9-f8c765ea3909-kube-api-access-6lz2l\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.444534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.433956    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-model-cache\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.444534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.434044    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-dshm\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.444534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.434101    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-tmp-dir\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.444534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.434124    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2f5787-d041-4361-aba9-f8c765ea3909-tls-certs\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.444534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.434169    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lz2l\" (UniqueName: \"kubernetes.io/projected/4a2f5787-d041-4361-aba9-f8c765ea3909-kube-api-access-6lz2l\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.444534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.434260    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-home\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.444534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.434729    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-home\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.456880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.453598    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-model-cache\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.471434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.457732    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-tmp-dir\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.483428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.478007    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-dshm\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.497417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.484534    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2f5787-d041-4361-aba9-f8c765ea3909-tls-certs\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.507046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.500051    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"]
Oct 30 21:15:29.529717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.528122    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lz2l\" (UniqueName: \"kubernetes.io/projected/4a2f5787-d041-4361-aba9-f8c765ea3909-kube-api-access-6lz2l\") pod \"distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") " pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.529993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.529974    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:29.562226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.560364    4918 generic.go:356] "Generic (PLEG): container finished" podID="c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f" containerID="e0dbc1e5c2ea3378b127061d625d1e356c8dd4b83f9c3cc7f7da128e7754ebb2" exitCode=0
Oct 30 21:15:29.562226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.562072    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2" event={"ID":"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f","Type":"ContainerDied","Data":"e0dbc1e5c2ea3378b127061d625d1e356c8dd4b83f9c3cc7f7da128e7754ebb2"}
Oct 30 21:15:29.565638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.565615    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:29.568245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.568222    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth" event={"ID":"0249779b-f7da-435d-ac3d-5c77766db493","Type":"ContainerDied","Data":"96f8e49f4edb9b7a6231cf96c1f5fc1a7699bbd2b92d89dfcca531fe874b50bb"}
Oct 30 21:15:29.568432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.568376    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f8e49f4edb9b7a6231cf96c1f5fc1a7699bbd2b92d89dfcca531fe874b50bb"
Oct 30 21:15:29.568532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.568232    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"
Oct 30 21:15:29.570184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.570129    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" event={"ID":"197109cc-f204-4d13-bf73-02172b3003ec","Type":"ContainerStarted","Data":"92454911f832d44911996bc3c7e62bc45b068b14d52175e06e70611130a9fff5"}
Oct 30 21:15:29.570184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.570180    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" event={"ID":"197109cc-f204-4d13-bf73-02172b3003ec","Type":"ContainerStarted","Data":"39e7f6ea793dfd2b980e487134be2d241494c33ed76e22b25fb9e86a075a484d"}
Oct 30 21:15:29.626796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.626763    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:29.637915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.637861    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" podStartSLOduration=2.637843658 podStartE2EDuration="2.637843658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:29.634832985 +0000 UTC m=+842.681392131" watchObservedRunningTime="2026-10-30 21:15:29.637843658 +0000 UTC m=+842.684402803"
Oct 30 21:15:29.864480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.863480    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"]
Oct 30 21:15:29.869489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.869460    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"]
Oct 30 21:15:29.869632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.869584    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:29.961352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.961327    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"]
Oct 30 21:15:29.961661 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:29.961636    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a2f5787_d041_4361_aba9_f8c765ea3909.slice/crio-0f3d7d9423f4e2619ab0613967728b5639b2abe1104f7305f7674dbac413f2ba WatchSource:0}: Error finding container 0f3d7d9423f4e2619ab0613967728b5639b2abe1104f7305f7674dbac413f2ba: Status 404 returned error can't find the container with id 0f3d7d9423f4e2619ab0613967728b5639b2abe1104f7305f7674dbac413f2ba
Oct 30 21:15:29.961890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.961871    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:29.961934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.961906    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:29.961970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.961941    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:29.962021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.962007    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:29.962100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.962078    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk6t2\" (UniqueName: \"kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-kube-api-access-fk6t2\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:29.962218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.962160    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:29.962261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.962221    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:29.962261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.962240    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dbe041-9629-4ee0-8221-79c215ac06d8-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:29.962318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:29.962307    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.063615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.063585    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.063764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.063747    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.063829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.063793    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.063892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.063836    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.063892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.063863    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.064006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.063902    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk6t2\" (UniqueName: \"kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-kube-api-access-fk6t2\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.064006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.063956    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.064107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.064020    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.064107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.064039    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dbe041-9629-4ee0-8221-79c215ac06d8-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.064213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.064117    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.064270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.064204    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.064559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.064532    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.064693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.064666    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dbe041-9629-4ee0-8221-79c215ac06d8-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.064941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.064707    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.066366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.066340    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.066754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.066731    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.076570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.076545    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.076683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.076644    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk6t2\" (UniqueName: \"kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-kube-api-access-fk6t2\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.185175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.185144    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:30.231431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.231339    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:30.233107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.233084    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:30.233203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.233132    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" podUID="197109cc-f204-4d13-bf73-02172b3003ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 21:15:30.518434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.517784    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"]
Oct 30 21:15:30.518758 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:30.518452    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71dbe041_9629_4ee0_8221_79c215ac06d8.slice/crio-ad9b442b742eabac582e53a2e0927fb9e5d892a65dea5a5a7a3f078e10f88ceb WatchSource:0}: Error finding container ad9b442b742eabac582e53a2e0927fb9e5d892a65dea5a5a7a3f078e10f88ceb: Status 404 returned error can't find the container with id ad9b442b742eabac582e53a2e0927fb9e5d892a65dea5a5a7a3f078e10f88ceb
Oct 30 21:15:30.521813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.521785    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:30.521976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.521961    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:30.522069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.522058    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:30.528538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.528519    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:30.582849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.582822    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" event={"ID":"71dbe041-9629-4ee0-8221-79c215ac06d8","Type":"ContainerStarted","Data":"ad9b442b742eabac582e53a2e0927fb9e5d892a65dea5a5a7a3f078e10f88ceb"}
Oct 30 21:15:30.584983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.584960    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" event={"ID":"4a2f5787-d041-4361-aba9-f8c765ea3909","Type":"ContainerStarted","Data":"c4f354300479e1180fac0f1195f25b0c107b127e9265702a990ba92dba149be0"}
Oct 30 21:15:30.585337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.584996    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" event={"ID":"4a2f5787-d041-4361-aba9-f8c765ea3909","Type":"ContainerStarted","Data":"0f3d7d9423f4e2619ab0613967728b5639b2abe1104f7305f7674dbac413f2ba"}
Oct 30 21:15:30.612610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.608929    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k" event={"ID":"2a85fae0-d8ef-4579-806b-54d744619594","Type":"ContainerStarted","Data":"7c56214b82c2d306af235b9c071b59e820e024bfd3c895c8e74dfe7edfef112b"}
Oct 30 21:15:30.612610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.608979    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k" event={"ID":"2a85fae0-d8ef-4579-806b-54d744619594","Type":"ContainerStarted","Data":"7b4735d4a4fbc2708352560d7f9dfa84d2401b17cc4a9a3351b3e5a05d2b12fe"}
Oct 30 21:15:30.627690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.627616    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:30.661158 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.661109    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podStartSLOduration=1.6610945 podStartE2EDuration="1.6610945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:30.635233113 +0000 UTC m=+843.681792288" watchObservedRunningTime="2026-10-30 21:15:30.6610945 +0000 UTC m=+843.707653690"
Oct 30 21:15:30.740454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.740106    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k" podStartSLOduration=2.7400919999999998 podStartE2EDuration="2.740092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:30.738571131 +0000 UTC m=+843.785130278" watchObservedRunningTime="2026-10-30 21:15:30.740092 +0000 UTC m=+843.786651144"
Oct 30 21:15:30.920876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:30.920841    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:31.080868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.080843    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-openshift-ca\") pod \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\" (UID: \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\") "
Oct 30 21:15:31.081052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.080891    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-kube-api-access-shxbs\" (UniqueName: \"kubernetes.io/projected/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-kube-api-access-shxbs\") pod \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\" (UID: \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\") "
Oct 30 21:15:31.081052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.080957    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-xks-ca\") pod \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\" (UID: \"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f\") "
Oct 30 21:15:31.081205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.081175    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-openshift-ca" pod "c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f" (UID: "c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:31.081520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.081497    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:31.083605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.083580    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-xks-ca" pod "c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f" (UID: "c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:31.083686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.083669    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-kube-api-access-shxbs" pod "c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f" (UID: "c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f"). InnerVolumeSpecName "kube-api-access-shxbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:31.182662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.182572    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shxbs\" (UniqueName: \"kubernetes.io/projected/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-kube-api-access-shxbs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:31.182662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.182609    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:31.234465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.234001    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:31.234465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.234063    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" podUID="197109cc-f204-4d13-bf73-02172b3003ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 21:15:31.640103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.639740    4918 generic.go:356] "Generic (PLEG): container finished" podID="2a85fae0-d8ef-4579-806b-54d744619594" containerID="7c56214b82c2d306af235b9c071b59e820e024bfd3c895c8e74dfe7edfef112b" exitCode=0
Oct 30 21:15:31.640103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.639789    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k" event={"ID":"2a85fae0-d8ef-4579-806b-54d744619594","Type":"ContainerDied","Data":"7c56214b82c2d306af235b9c071b59e820e024bfd3c895c8e74dfe7edfef112b"}
Oct 30 21:15:31.641466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.641439    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"
Oct 30 21:15:31.641576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.641466    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2" event={"ID":"c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f","Type":"ContainerDied","Data":"793fc4c2c8955ec3fd54c13ebe9cc0b5f0f822c78c8c26ffc6ffd26be63d0916"}
Oct 30 21:15:31.641576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.641505    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="793fc4c2c8955ec3fd54c13ebe9cc0b5f0f822c78c8c26ffc6ffd26be63d0916"
Oct 30 21:15:31.643724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.643697    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" event={"ID":"71dbe041-9629-4ee0-8221-79c215ac06d8","Type":"ContainerStarted","Data":"fbbcaa1419f9dc011f5101982a6e6d57d236cd603d26f4e845e8fc4663bce946"}
Oct 30 21:15:31.742181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:31.742122    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" podStartSLOduration=2.742106587 podStartE2EDuration="2.742106587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:31.704866341 +0000 UTC m=+844.751425519" watchObservedRunningTime="2026-10-30 21:15:31.742106587 +0000 UTC m=+844.788665731"
Oct 30 21:15:32.198418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:32.194689    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:32.203827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:32.203495    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:32.203827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:32.203554    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" podUID="71dbe041-9629-4ee0-8221-79c215ac06d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 21:15:32.234275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:32.233832    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:32.234275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:32.233892    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" podUID="197109cc-f204-4d13-bf73-02172b3003ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 21:15:32.983981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:32.983959    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:33.049245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.049212    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a85fae0-d8ef-4579-806b-54d744619594-kube-api-access-wjzqg\" (UniqueName: \"kubernetes.io/projected/2a85fae0-d8ef-4579-806b-54d744619594-kube-api-access-wjzqg\") pod \"2a85fae0-d8ef-4579-806b-54d744619594\" (UID: \"2a85fae0-d8ef-4579-806b-54d744619594\") "
Oct 30 21:15:33.049491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.049454    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-xks-ca\") pod \"2a85fae0-d8ef-4579-806b-54d744619594\" (UID: \"2a85fae0-d8ef-4579-806b-54d744619594\") "
Oct 30 21:15:33.049646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.049560    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-openshift-ca\") pod \"2a85fae0-d8ef-4579-806b-54d744619594\" (UID: \"2a85fae0-d8ef-4579-806b-54d744619594\") "
Oct 30 21:15:33.050630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.050317    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-openshift-ca" pod "2a85fae0-d8ef-4579-806b-54d744619594" (UID: "2a85fae0-d8ef-4579-806b-54d744619594"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:33.052787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.052764    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a85fae0-d8ef-4579-806b-54d744619594-kube-api-access-wjzqg" pod "2a85fae0-d8ef-4579-806b-54d744619594" (UID: "2a85fae0-d8ef-4579-806b-54d744619594"). InnerVolumeSpecName "kube-api-access-wjzqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:33.053090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.053065    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-xks-ca" pod "2a85fae0-d8ef-4579-806b-54d744619594" (UID: "2a85fae0-d8ef-4579-806b-54d744619594"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:33.150822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.150783    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:33.150822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.150815    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjzqg\" (UniqueName: \"kubernetes.io/projected/2a85fae0-d8ef-4579-806b-54d744619594-kube-api-access-wjzqg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:33.150986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.150829    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a85fae0-d8ef-4579-806b-54d744619594-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:33.214476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.210853    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:33.214476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.210909    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" podUID="71dbe041-9629-4ee0-8221-79c215ac06d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 21:15:33.240134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.239942    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:33.240134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.240007    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" podUID="197109cc-f204-4d13-bf73-02172b3003ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 21:15:33.659284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.659253    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"
Oct 30 21:15:33.659284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.659276    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k" event={"ID":"2a85fae0-d8ef-4579-806b-54d744619594","Type":"ContainerDied","Data":"7b4735d4a4fbc2708352560d7f9dfa84d2401b17cc4a9a3351b3e5a05d2b12fe"}
Oct 30 21:15:33.659581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:33.659336    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b4735d4a4fbc2708352560d7f9dfa84d2401b17cc4a9a3351b3e5a05d2b12fe"
Oct 30 21:15:34.189415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.187727    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:34.189415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.187782    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" podUID="71dbe041-9629-4ee0-8221-79c215ac06d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 21:15:34.236414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.235707    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:34.236414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.235767    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" podUID="197109cc-f204-4d13-bf73-02172b3003ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 21:15:34.639321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.639288    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"]
Oct 30 21:15:34.639858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.639842    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2a85fae0-d8ef-4579-806b-54d744619594" containerName="revoke-keys"
Oct 30 21:15:34.639925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.639861    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a85fae0-d8ef-4579-806b-54d744619594" containerName="revoke-keys"
Oct 30 21:15:34.639925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.639885    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f" containerName="revoke-keys"
Oct 30 21:15:34.639925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.639891    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f" containerName="revoke-keys"
Oct 30 21:15:34.640020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.639982    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a85fae0-d8ef-4579-806b-54d744619594" containerName="revoke-keys"
Oct 30 21:15:34.640020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.639992    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f" containerName="revoke-keys"
Oct 30 21:15:34.643566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.643546    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.670581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.670551    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.670822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.670802    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.670965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.670950    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.671073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.671058    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.671177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.671164    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.671310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.671296    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.671524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.671508    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.671806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.671753    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.671911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.671891    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdpt8\" (UniqueName: \"kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-kube-api-access-rdpt8\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.716422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.682666    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"]
Oct 30 21:15:34.824668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.823636    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.824668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.823697    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.824668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.823747    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.824668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.823824    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdpt8\" (UniqueName: \"kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-kube-api-access-rdpt8\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.824668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.823880    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.824668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.823925    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.824668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.823988    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.824668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.824018    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.824668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.824056    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.825360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.824933    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.825360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.825341    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.825530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.825521    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.825891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.825869    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.826325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.826306    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.828694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.828667    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.829177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.829156    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.835412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.835333    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:34.835830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:34.835809    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdpt8\" (UniqueName: \"kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-kube-api-access-rdpt8\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:35.004972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.003624    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:35.191416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.190756    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:35.191416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.190811    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" podUID="71dbe041-9629-4ee0-8221-79c215ac06d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 21:15:35.231159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.231132    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 30 21:15:35.231305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.231181    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" podUID="197109cc-f204-4d13-bf73-02172b3003ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 30 21:15:35.317671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.317634    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"]
Oct 30 21:15:35.318851 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:35.318821    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd9f7af_1f13_4301_b393_fe62a8029ea2.slice/crio-eeafc86573e42ecf96b2406c374a8259d1112cf7175584ac119932d8c1a04dcc WatchSource:0}: Error finding container eeafc86573e42ecf96b2406c374a8259d1112cf7175584ac119932d8c1a04dcc: Status 404 returned error can't find the container with id eeafc86573e42ecf96b2406c374a8259d1112cf7175584ac119932d8c1a04dcc
Oct 30 21:15:35.321923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.321893    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:35.322013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.321962    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:35.322013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.321996    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:35.368434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.368409    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"]
Oct 30 21:15:35.373750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.373729    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:35.379067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.378798    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:35.379067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.378838    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:15:35.380603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.379611    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:15:35.380603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.380149    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"]
Oct 30 21:15:35.431654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.431628    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdq5f\" (UniqueName: \"kubernetes.io/projected/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-kube-api-access-wdq5f\") pod \"maas-api-revoke-sub-972c9a4c-67946\" (UID: \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:35.431910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.431895    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972c9a4c-67946\" (UID: \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:35.432009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.431999    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972c9a4c-67946\" (UID: \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:35.480978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.478123    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:15:35.533053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.533023    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972c9a4c-67946\" (UID: \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:35.533249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.533065    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972c9a4c-67946\" (UID: \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:35.533249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.533142    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdq5f\" (UniqueName: \"kubernetes.io/projected/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-kube-api-access-wdq5f\") pod \"maas-api-revoke-sub-972c9a4c-67946\" (UID: \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:35.533849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.533816    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-972c9a4c-67946\" (UID: \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:35.535765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.535742    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-972c9a4c-67946\" (UID: \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:35.542997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.542969    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdq5f\" (UniqueName: \"kubernetes.io/projected/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-kube-api-access-wdq5f\") pod \"maas-api-revoke-sub-972c9a4c-67946\" (UID: \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:35.726173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.726144    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:35.727093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.727057    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" event={"ID":"dfd9f7af-1f13-4301-b393-fe62a8029ea2","Type":"ContainerStarted","Data":"a727d1b8a70d250b05240227fdb6ea7729f15ae9b5034fec1f31eea96d1623af"}
Oct 30 21:15:35.727189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.727143    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" event={"ID":"dfd9f7af-1f13-4301-b393-fe62a8029ea2","Type":"ContainerStarted","Data":"eeafc86573e42ecf96b2406c374a8259d1112cf7175584ac119932d8c1a04dcc"}
Oct 30 21:15:35.750067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:35.750022    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podStartSLOduration=1.750007745 podStartE2EDuration="1.750007745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:35.747097082 +0000 UTC m=+848.793656253" watchObservedRunningTime="2026-10-30 21:15:35.750007745 +0000 UTC m=+848.796566890"
Oct 30 21:15:36.008108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:36.007975    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:36.008747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:36.008376    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:36.008747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:36.008449    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 21:15:36.042173 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:36.042137    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b116033_6890_4cb3_b1b3_ff8e8b25a8c0.slice/crio-1db8d07a6f5d0668f26d55fbff885ab0a7d4709acc3ede1dcb48611b52d457c7 WatchSource:0}: Error finding container 1db8d07a6f5d0668f26d55fbff885ab0a7d4709acc3ede1dcb48611b52d457c7: Status 404 returned error can't find the container with id 1db8d07a6f5d0668f26d55fbff885ab0a7d4709acc3ede1dcb48611b52d457c7
Oct 30 21:15:36.044208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:36.044153    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"]
Oct 30 21:15:36.185856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:36.185815    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:36.186155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:36.186115    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" podUID="71dbe041-9629-4ee0-8221-79c215ac06d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 21:15:36.547375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:36.547347    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:36.548710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:36.548675    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:36.549993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:36.549976    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:15:36.766623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:36.766578    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946" event={"ID":"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0","Type":"ContainerStarted","Data":"4b5c203196f232b734b262d7debfff1df2727cf9e842d97f5ed6a6ab1bc0d93e"}
Oct 30 21:15:36.766790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:36.766654    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946" event={"ID":"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0","Type":"ContainerStarted","Data":"1db8d07a6f5d0668f26d55fbff885ab0a7d4709acc3ede1dcb48611b52d457c7"}
Oct 30 21:15:36.824416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:36.822733    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946" podStartSLOduration=1.822708888 podStartE2EDuration="1.822708888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:36.784764287 +0000 UTC m=+849.831323453" watchObservedRunningTime="2026-10-30 21:15:36.822708888 +0000 UTC m=+849.869268034"
Oct 30 21:15:37.006465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:37.005796    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:37.006465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:37.005873    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 21:15:37.185615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:37.185534    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:15:37.187450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:37.185814    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" podUID="71dbe041-9629-4ee0-8221-79c215ac06d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 21:15:37.775555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:37.775522    4918 generic.go:356] "Generic (PLEG): container finished" podID="1b116033-6890-4cb3-b1b3-ff8e8b25a8c0" containerID="4b5c203196f232b734b262d7debfff1df2727cf9e842d97f5ed6a6ab1bc0d93e" exitCode=0
Oct 30 21:15:37.776047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:37.775613    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946" event={"ID":"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0","Type":"ContainerDied","Data":"4b5c203196f232b734b262d7debfff1df2727cf9e842d97f5ed6a6ab1bc0d93e"}
Oct 30 21:15:38.030490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.027564    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:38.030490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.027627    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 21:15:38.208696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.208664    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:38.210324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.210304    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:38.221415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.218673    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:15:38.425468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.425261    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"]
Oct 30 21:15:38.431264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.430497    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.431895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.431872    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"]
Oct 30 21:15:38.437092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.437071    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.459005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.458886    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"]
Oct 30 21:15:38.469987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.469959    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"]
Oct 30 21:15:38.475005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.474978    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.475122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.475013    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.475208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.475121    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.475208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.475146    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.475322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.475239    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.475322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.475255    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.475322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.475290    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.475322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.475319    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.475592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.475335    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5hp\" (UniqueName: \"kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-kube-api-access-rf5hp\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.576437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.576372    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.576610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.576445    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.576610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.576520    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.576610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.576542    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0af372e-9d87-4659-ac38-63117bc6321b-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.576610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.576566    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.576610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.576602    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0af372e-9d87-4659-ac38-63117bc6321b-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.576844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.576711    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2c5q\" (UniqueName: \"kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-kube-api-access-b2c5q\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.576844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.576800    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.576905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.576837    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.576905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.576890    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.577002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.576988    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.577034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.577010    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.577068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.577030    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.577068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.577050    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.577132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.577117    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.577167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.577133    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.577209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.577167    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.577263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.577235    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.577263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.577238    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.577368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.577259    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5hp\" (UniqueName: \"kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-kube-api-access-rf5hp\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.577368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.577286    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.577509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.577449    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.577589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.577567    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.579469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.579449    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.579690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.579673    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.610417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.585779    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.610417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.603032    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5hp\" (UniqueName: \"kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-kube-api-access-rf5hp\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.678606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.678532    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.678606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.678577    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.678808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.678608    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.678808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.678695    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0af372e-9d87-4659-ac38-63117bc6321b-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.678808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.678732    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0af372e-9d87-4659-ac38-63117bc6321b-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.678808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.678754    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2c5q\" (UniqueName: \"kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-kube-api-access-b2c5q\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.678808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.678789    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.679058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.678967    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-data\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.679112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.679050    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-credential-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.679257 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.679236    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-certs\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.679307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.679298    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.679344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.679319    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0af372e-9d87-4659-ac38-63117bc6321b-istiod-ca-cert\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.679344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.679321    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.679555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.679539    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-socket\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.681382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.681361    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0af372e-9d87-4659-ac38-63117bc6321b-istio-podinfo\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.681740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.681719    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-envoy\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.714854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.714826    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-istio-token\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.716031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.716000    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2c5q\" (UniqueName: \"kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-kube-api-access-b2c5q\") pod \"e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") " pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:38.786428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.784174    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:38.801980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:38.801669    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:39.008520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.005805    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:39.008520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.005895    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 21:15:39.179177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.179151    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:39.325456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.324658    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-xks-ca\") pod \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\" (UID: \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\") "
Oct 30 21:15:39.325456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.324858    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-openshift-ca\") pod \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\" (UID: \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\") "
Oct 30 21:15:39.325456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.324988    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-kube-api-access-wdq5f\" (UniqueName: \"kubernetes.io/projected/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-kube-api-access-wdq5f\") pod \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\" (UID: \"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0\") "
Oct 30 21:15:39.347862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.347074    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-openshift-ca" pod "1b116033-6890-4cb3-b1b3-ff8e8b25a8c0" (UID: "1b116033-6890-4cb3-b1b3-ff8e8b25a8c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:39.349378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.349340    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-xks-ca" pod "1b116033-6890-4cb3-b1b3-ff8e8b25a8c0" (UID: "1b116033-6890-4cb3-b1b3-ff8e8b25a8c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:39.350663 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:39.350628    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0af372e_9d87_4659_ac38_63117bc6321b.slice/crio-bdccd95a2f8b70e5e971c4aaa8dda5ed1febc425efa1b23d738502dbe7c19632 WatchSource:0}: Error finding container bdccd95a2f8b70e5e971c4aaa8dda5ed1febc425efa1b23d738502dbe7c19632: Status 404 returned error can't find the container with id bdccd95a2f8b70e5e971c4aaa8dda5ed1febc425efa1b23d738502dbe7c19632
Oct 30 21:15:39.351553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.351513    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"]
Oct 30 21:15:39.355312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.353782    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-kube-api-access-wdq5f" pod "1b116033-6890-4cb3-b1b3-ff8e8b25a8c0" (UID: "1b116033-6890-4cb3-b1b3-ff8e8b25a8c0"). InnerVolumeSpecName "kube-api-access-wdq5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:39.355312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.354377    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:39.355312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.354478    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:39.355312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.354520    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:39.355312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.354714    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:39.355312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.354789    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:39.355312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.354853    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:39.361112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.361090    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"]
Oct 30 21:15:39.428938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.428912    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdq5f\" (UniqueName: \"kubernetes.io/projected/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-kube-api-access-wdq5f\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:39.428938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.428938    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:39.429091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.428948    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:39.628849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.627447    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:39.643863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.643838    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:39.857513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.857428    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" event={"ID":"b0af372e-9d87-4659-ac38-63117bc6321b","Type":"ContainerStarted","Data":"82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528"}
Oct 30 21:15:39.857513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.857493    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" event={"ID":"b0af372e-9d87-4659-ac38-63117bc6321b","Type":"ContainerStarted","Data":"bdccd95a2f8b70e5e971c4aaa8dda5ed1febc425efa1b23d738502dbe7c19632"}
Oct 30 21:15:39.859869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.859844    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" event={"ID":"d919503b-cf6f-4c14-8a83-167f5c35f2cd","Type":"ContainerStarted","Data":"628b293d70552b7591bf94a822b9c14394cc0dba5f8aa6678bd9c9dfd42c1286"}
Oct 30 21:15:39.859982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.859893    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" event={"ID":"d919503b-cf6f-4c14-8a83-167f5c35f2cd","Type":"ContainerStarted","Data":"30c7ce2d3026057cfd46eac6cb2bb2376a2787418c762025de404f0ee8b21a21"}
Oct 30 21:15:39.861131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.861110    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"
Oct 30 21:15:39.861202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.861130    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946" event={"ID":"1b116033-6890-4cb3-b1b3-ff8e8b25a8c0","Type":"ContainerDied","Data":"1db8d07a6f5d0668f26d55fbff885ab0a7d4709acc3ede1dcb48611b52d457c7"}
Oct 30 21:15:39.861202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.861172    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db8d07a6f5d0668f26d55fbff885ab0a7d4709acc3ede1dcb48611b52d457c7"
Oct 30 21:15:39.873712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.873692    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:39.907413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.902059    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podStartSLOduration=1.902037339 podStartE2EDuration="1.902037339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:39.879554878 +0000 UTC m=+852.926114041" watchObservedRunningTime="2026-10-30 21:15:39.902037339 +0000 UTC m=+852.948596485"
Oct 30 21:15:39.929410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:39.927310    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podStartSLOduration=1.9272960000000001 podStartE2EDuration="1.927296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:39.918868141 +0000 UTC m=+852.965427289" watchObservedRunningTime="2026-10-30 21:15:39.927296 +0000 UTC m=+852.973855210"
Oct 30 21:15:40.008416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:40.004913    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:40.008416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:40.005001    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 21:15:40.793215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:40.791773    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:40.806418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:40.805530    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:15:40.806418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:40.805593    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 21:15:40.808408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:40.808362    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:40.809686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:40.809663    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:15:40.809887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:40.809867    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:15:41.020417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:41.010508    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:41.020417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:41.010569    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 21:15:41.789414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:41.788780    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:15:41.789414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:41.788844    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 21:15:41.809417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:41.805758    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:15:41.809417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:41.805815    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:15:42.016413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.015864    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:42.016413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.015928    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 21:15:42.069528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.067573    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:15:42.160790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.159511    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:42.257285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.255467    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:15:42.301934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.301890    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:42.342966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.342863    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:42.346301 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.344851    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:42.403421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.402664    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:42.465525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.465478    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:15:42.516526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.516470    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:15:42.563907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.563860    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:42.793816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.793777    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:15:42.794132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.794106    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 21:15:42.807562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.807520    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:15:42.807822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.807796    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:15:42.866565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:42.866524    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.007417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.004933    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:43.007417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.004993    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 21:15:43.072497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.068477    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.072497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.069105    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:15:43.161538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.161485    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.293450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.287847    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.324469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.322922    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.419322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.406527    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.548414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.436344    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.549963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.466521    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:15:43.552742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.551738    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:15:43.552742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.551804    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.552742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.518321    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.569414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.492938    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.571461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.571427    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:43.600695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.599846    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:15:43.600969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.600954    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:15:43.601136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.601124    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:15:43.601305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.601293    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:15:43.601478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.601466    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:15:43.601633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.601621    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:15:43.601782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.601772    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:43.601940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.601928    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:15:43.602076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.602065    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:15:43.943576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.931610    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.123445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.944613    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:15:44.123445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.944667    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:15:44.123445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.944761    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:44.123445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.950305    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:15:44.123445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:43.950346    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 21:15:44.123445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.008490    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:44.123445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.008553    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 21:15:44.123445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.082780    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:44.200101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.199731    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.592593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.567923    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.605632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.593574    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:44.605632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.595195    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:44.605632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.595563    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:44.605632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.595942    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.605632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.596269    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.605632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.605526    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:15:44.633425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.621543    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:44.645600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.645561    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:44.795417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.792223    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:15:45.261654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.261612    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 21:15:45.287476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.131460    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.287715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.956500    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.313938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:44.865982    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:15:45.314199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.314166    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:15:45.314409 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.279653    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.314635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.028319    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:45.314755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.314737    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 21:15:45.603468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.581169    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:15:45.634451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.634411    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.662118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.660118    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.683093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.663316    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.683093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.663430    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.684071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.684027    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.684617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.684590    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:15:45.685033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.685010    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:15:45.685424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.685378    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:15:45.729906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.729855    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 21:15:45.787434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.787365    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:45.877014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.876963    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:15:45.877270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.877241    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:15:45.882472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.880813    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:15:45.882472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.880871    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 21:15:45.895136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:45.894152    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:15:46.031427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.026296    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:15:46.052082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.052029    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 21:15:46.052354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.032316    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Oct 30 21:15:46.368416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.326602    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:46.423713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.423671    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:15:46.606506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.606097    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:46.635331 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.635222    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:15:46.670417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.662155    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:46.670717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.670684    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:46.700329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.699060    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:46.700329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.699441    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:15:46.700329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.699722    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:15:46.741102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.740516    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:46.795237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.795198    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:15:46.810499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.810455    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 21:15:46.811764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.811733    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:15:46.811941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.811917    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:15:46.974585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:46.954425    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:15:47.183780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.183746    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:15:47.183946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.183820    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:15:47.333850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.329882    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:47.443417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.435836    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:47.450747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.450720    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:47.465376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.465340    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:15:47.471598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.471574    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:15:47.499931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.497722    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:47.543199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.543171    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:15:47.581200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.581173    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:15:47.649475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.641564    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:47.682255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.671701    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:47.709540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.700555    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:15:47.742527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.742488    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:15:47.828416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.786883    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:15:47.854416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.817931    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:15:47.854416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.839537    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 21:15:47.854416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.820239    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:15:47.854416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.839936    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:15:47.868720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.855780    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:15:47.868720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.857741    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:15:47.876178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.876159    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:15:47.882979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:47.882316    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:15:48.814414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:48.807775    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:15:48.814414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:48.807837    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 21:15:48.814414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:48.808721    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:15:48.814414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:48.808764    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:15:49.569997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.569964    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"]
Oct 30 21:15:49.570601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.570584    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1b116033-6890-4cb3-b1b3-ff8e8b25a8c0" containerName="revoke-keys"
Oct 30 21:15:49.570601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.570601    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b116033-6890-4cb3-b1b3-ff8e8b25a8c0" containerName="revoke-keys"
Oct 30 21:15:49.570735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.570692    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b116033-6890-4cb3-b1b3-ff8e8b25a8c0" containerName="revoke-keys"
Oct 30 21:15:49.576017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.575986    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:49.622434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.585258    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:15:49.622434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.585283    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:15:49.622434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.585373    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:15:49.622434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.586211    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"]
Oct 30 21:15:49.622434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.598655    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:15:49.649587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.649548    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53582c8-7b6gl\" (UID: \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:49.649778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.649700    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zkm8\" (UniqueName: \"kubernetes.io/projected/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-kube-api-access-2zkm8\") pod \"maas-api-revoke-sub-a53582c8-7b6gl\" (UID: \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:49.649778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.649770    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53582c8-7b6gl\" (UID: \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:49.680417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.676433    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"]
Oct 30 21:15:49.681325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.681306    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.732981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.732945    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"]
Oct 30 21:15:49.750851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.750817    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53582c8-7b6gl\" (UID: \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:49.751068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.751050    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.751232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.751218    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f719e78-068e-4c52-96c5-58eddc3de968-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.751339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.751327    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f719e78-068e-4c52-96c5-58eddc3de968-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.751454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.751441    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zkm8\" (UniqueName: \"kubernetes.io/projected/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-kube-api-access-2zkm8\") pod \"maas-api-revoke-sub-a53582c8-7b6gl\" (UID: \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:49.751547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.751524    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53582c8-7b6gl\" (UID: \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:49.751772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.751577    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53582c8-7b6gl\" (UID: \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:49.751874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.751857    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.751958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.751941    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.752016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.751981    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.752080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.752018    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.752130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.752116    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbml7\" (UniqueName: \"kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-kube-api-access-tbml7\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.752186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.752148    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.754911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.754843    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53582c8-7b6gl\" (UID: \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:49.783115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.783085    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zkm8\" (UniqueName: \"kubernetes.io/projected/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-kube-api-access-2zkm8\") pod \"maas-api-revoke-sub-a53582c8-7b6gl\" (UID: \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:49.788328 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.788302    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 21:15:49.788471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.788367    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 21:15:49.803527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.803504    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:15:49.803656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.803543    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:15:49.852903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.852821    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.852903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.852858    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.852903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.852877    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.852903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.852897    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.853520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.852996    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbml7\" (UniqueName: \"kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-kube-api-access-tbml7\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.853520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.853025    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.853520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.853074    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.853520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.853164    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f719e78-068e-4c52-96c5-58eddc3de968-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.853520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.853189    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f719e78-068e-4c52-96c5-58eddc3de968-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.853520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.853234    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.853520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.853283    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.853897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.853615    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.853897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.853830    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.854033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.854013    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f719e78-068e-4c52-96c5-58eddc3de968-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.855424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.855376    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.856115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.856092    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f719e78-068e-4c52-96c5-58eddc3de968-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.877745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.877715    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.878591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.878570    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbml7\" (UniqueName: \"kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-kube-api-access-tbml7\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:49.943574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:49.943548    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:50.036537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:50.036198    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:50.233057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:50.233027    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"]
Oct 30 21:15:50.233941 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:50.233912    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fcc7dd_87e0_4995_82bd_a6b55ed5dd95.slice/crio-727feb39925ce97e9f36ea0b539a1daab8439f39e0d5b98589f4502f46adc38e WatchSource:0}: Error finding container 727feb39925ce97e9f36ea0b539a1daab8439f39e0d5b98589f4502f46adc38e: Status 404 returned error can't find the container with id 727feb39925ce97e9f36ea0b539a1daab8439f39e0d5b98589f4502f46adc38e
Oct 30 21:15:50.327488 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:50.327458    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f719e78_068e_4c52_96c5_58eddc3de968.slice/crio-e9879d248f6094918389effbee441d6371327ce09625f43417add182ac877cf7 WatchSource:0}: Error finding container e9879d248f6094918389effbee441d6371327ce09625f43417add182ac877cf7: Status 404 returned error can't find the container with id e9879d248f6094918389effbee441d6371327ce09625f43417add182ac877cf7
Oct 30 21:15:50.329379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:50.329351    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"]
Oct 30 21:15:50.330600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:50.330563    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:50.330680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:50.330646    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:50.330715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:50.330689    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:15:51.081285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:51.081128    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" event={"ID":"5f719e78-068e-4c52-96c5-58eddc3de968","Type":"ContainerStarted","Data":"6969b30b94575aaf43fa8c16fed170bbef6b68dccc1546a08d7fb450c3ec6bc1"}
Oct 30 21:15:51.081285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:51.081195    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" event={"ID":"5f719e78-068e-4c52-96c5-58eddc3de968","Type":"ContainerStarted","Data":"e9879d248f6094918389effbee441d6371327ce09625f43417add182ac877cf7"}
Oct 30 21:15:51.081285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:51.081222    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl" event={"ID":"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95","Type":"ContainerStarted","Data":"4f72bd9a1161f85e8c7a015c314b29d4641c8ccdd4e52467e3c6fc86d8904e51"}
Oct 30 21:15:51.081285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:51.081245    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl" event={"ID":"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95","Type":"ContainerStarted","Data":"727feb39925ce97e9f36ea0b539a1daab8439f39e0d5b98589f4502f46adc38e"}
Oct 30 21:15:51.137427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:51.114516    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" podStartSLOduration=2.114491046 podStartE2EDuration="2.114491046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:51.111885623 +0000 UTC m=+864.158444769" watchObservedRunningTime="2026-10-30 21:15:51.114491046 +0000 UTC m=+864.161050273"
Oct 30 21:15:51.144977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:51.144923    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl" podStartSLOduration=2.144909523 podStartE2EDuration="2.144909523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:51.14290138 +0000 UTC m=+864.189460525" watchObservedRunningTime="2026-10-30 21:15:51.144909523 +0000 UTC m=+864.191468668"
Oct 30 21:15:51.449812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:51.449784    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:51.450555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:51.450527    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:51.451669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:51.451649    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:15:51.453862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:51.453806    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:51.454175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:51.454156    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:51.456084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:51.456063    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:15:52.036988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.036960    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:52.038728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.038699    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:15:52.038828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.038754    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" podUID="5f719e78-068e-4c52-96c5-58eddc3de968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 21:15:52.112639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.112604    4918 generic.go:356] "Generic (PLEG): container finished" podID="18fcc7dd-87e0-4995-82bd-a6b55ed5dd95" containerID="4f72bd9a1161f85e8c7a015c314b29d4641c8ccdd4e52467e3c6fc86d8904e51" exitCode=0
Oct 30 21:15:52.113044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.112694    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl" event={"ID":"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95","Type":"ContainerDied","Data":"4f72bd9a1161f85e8c7a015c314b29d4641c8ccdd4e52467e3c6fc86d8904e51"}
Oct 30 21:15:52.218752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.218719    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"]
Oct 30 21:15:52.223282 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.223117    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:52.226102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.226076    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"]
Oct 30 21:15:52.420076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.419977    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8402af68-2zbcn\" (UID: \"a56a6e36-5652-409e-b2f0-e912f657c8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:52.420224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.420191    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wl9\" (UniqueName: \"kubernetes.io/projected/a56a6e36-5652-409e-b2f0-e912f657c8c6-kube-api-access-t8wl9\") pod \"maas-api-revoke-sub-8402af68-2zbcn\" (UID: \"a56a6e36-5652-409e-b2f0-e912f657c8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:52.420284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.420243    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8402af68-2zbcn\" (UID: \"a56a6e36-5652-409e-b2f0-e912f657c8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:52.522081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.522048    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wl9\" (UniqueName: \"kubernetes.io/projected/a56a6e36-5652-409e-b2f0-e912f657c8c6-kube-api-access-t8wl9\") pod \"maas-api-revoke-sub-8402af68-2zbcn\" (UID: \"a56a6e36-5652-409e-b2f0-e912f657c8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:52.522274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.522102    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8402af68-2zbcn\" (UID: \"a56a6e36-5652-409e-b2f0-e912f657c8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:52.522274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.522174    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8402af68-2zbcn\" (UID: \"a56a6e36-5652-409e-b2f0-e912f657c8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:52.522972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.522888    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8402af68-2zbcn\" (UID: \"a56a6e36-5652-409e-b2f0-e912f657c8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:52.524637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.524616    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8402af68-2zbcn\" (UID: \"a56a6e36-5652-409e-b2f0-e912f657c8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:52.530620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.530602    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wl9\" (UniqueName: \"kubernetes.io/projected/a56a6e36-5652-409e-b2f0-e912f657c8c6-kube-api-access-t8wl9\") pod \"maas-api-revoke-sub-8402af68-2zbcn\" (UID: \"a56a6e36-5652-409e-b2f0-e912f657c8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:52.549210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.549192    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:52.865592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:52.865569    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"]
Oct 30 21:15:52.866160 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:52.866136    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56a6e36_5652_409e_b2f0_e912f657c8c6.slice/crio-536e1175daa0ded30641dfeff7e493b8834924f981237112a780ff5af5ffce58 WatchSource:0}: Error finding container 536e1175daa0ded30641dfeff7e493b8834924f981237112a780ff5af5ffce58: Status 404 returned error can't find the container with id 536e1175daa0ded30641dfeff7e493b8834924f981237112a780ff5af5ffce58
Oct 30 21:15:53.038175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.038133    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:15:53.038383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.038200    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" podUID="5f719e78-068e-4c52-96c5-58eddc3de968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 21:15:53.120572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.120537    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn" event={"ID":"a56a6e36-5652-409e-b2f0-e912f657c8c6","Type":"ContainerStarted","Data":"536e1175daa0ded30641dfeff7e493b8834924f981237112a780ff5af5ffce58"}
Oct 30 21:15:53.281454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.281430    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:53.358559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.358535    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-kube-api-access-2zkm8\" (UniqueName: \"kubernetes.io/projected/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-kube-api-access-2zkm8\") pod \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\" (UID: \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\") "
Oct 30 21:15:53.358818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.358638    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-xks-ca\") pod \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\" (UID: \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\") "
Oct 30 21:15:53.358895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.358831    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-openshift-ca\") pod \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\" (UID: \"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95\") "
Oct 30 21:15:53.359193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.359168    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-openshift-ca" pod "18fcc7dd-87e0-4995-82bd-a6b55ed5dd95" (UID: "18fcc7dd-87e0-4995-82bd-a6b55ed5dd95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:53.360993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.360973    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-kube-api-access-2zkm8" pod "18fcc7dd-87e0-4995-82bd-a6b55ed5dd95" (UID: "18fcc7dd-87e0-4995-82bd-a6b55ed5dd95"). InnerVolumeSpecName "kube-api-access-2zkm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:53.361206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.361186    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-xks-ca" pod "18fcc7dd-87e0-4995-82bd-a6b55ed5dd95" (UID: "18fcc7dd-87e0-4995-82bd-a6b55ed5dd95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:53.460360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.460327    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zkm8\" (UniqueName: \"kubernetes.io/projected/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-kube-api-access-2zkm8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:53.460360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.460356    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:53.460360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:53.460367    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:54.037114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.037086    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:15:54.037255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.037136    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" podUID="5f719e78-068e-4c52-96c5-58eddc3de968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 21:15:54.129189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.129151    4918 generic.go:356] "Generic (PLEG): container finished" podID="a56a6e36-5652-409e-b2f0-e912f657c8c6" containerID="0e84ba44279e00c4bf0c5b8e7da3a7daa9850fc08ea53a2326cd7d4a05e0a854" exitCode=0
Oct 30 21:15:54.129772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.129236    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn" event={"ID":"a56a6e36-5652-409e-b2f0-e912f657c8c6","Type":"ContainerDied","Data":"0e84ba44279e00c4bf0c5b8e7da3a7daa9850fc08ea53a2326cd7d4a05e0a854"}
Oct 30 21:15:54.130734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.130717    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"
Oct 30 21:15:54.130828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.130749    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl" event={"ID":"18fcc7dd-87e0-4995-82bd-a6b55ed5dd95","Type":"ContainerDied","Data":"727feb39925ce97e9f36ea0b539a1daab8439f39e0d5b98589f4502f46adc38e"}
Oct 30 21:15:54.130828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.130790    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="727feb39925ce97e9f36ea0b539a1daab8439f39e0d5b98589f4502f46adc38e"
Oct 30 21:15:54.637332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.637301    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"]
Oct 30 21:15:54.637938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.637922    4918 cpu_manager.go:414] "containerMap: removing container" podUID="18fcc7dd-87e0-4995-82bd-a6b55ed5dd95" containerName="revoke-keys"
Oct 30 21:15:54.637993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.637940    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18fcc7dd-87e0-4995-82bd-a6b55ed5dd95" containerName="revoke-keys"
Oct 30 21:15:54.638059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.638050    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="18fcc7dd-87e0-4995-82bd-a6b55ed5dd95" containerName="revoke-keys"
Oct 30 21:15:54.641601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.641586    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:54.651905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.651883    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"]
Oct 30 21:15:54.727311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.727264    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qdqt\" (UniqueName: \"kubernetes.io/projected/f5212a51-1dab-44ae-9c46-2ca807ab7143-kube-api-access-8qdqt\") pod \"maas-api-revoke-sub-9a3caf3a-s6zmn\" (UID: \"f5212a51-1dab-44ae-9c46-2ca807ab7143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:54.728214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.728186    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a3caf3a-s6zmn\" (UID: \"f5212a51-1dab-44ae-9c46-2ca807ab7143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:54.728549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.728476    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a3caf3a-s6zmn\" (UID: \"f5212a51-1dab-44ae-9c46-2ca807ab7143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:54.830491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.830462    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a3caf3a-s6zmn\" (UID: \"f5212a51-1dab-44ae-9c46-2ca807ab7143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:54.830705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.830500    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a3caf3a-s6zmn\" (UID: \"f5212a51-1dab-44ae-9c46-2ca807ab7143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:54.830783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.830717    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qdqt\" (UniqueName: \"kubernetes.io/projected/f5212a51-1dab-44ae-9c46-2ca807ab7143-kube-api-access-8qdqt\") pod \"maas-api-revoke-sub-9a3caf3a-s6zmn\" (UID: \"f5212a51-1dab-44ae-9c46-2ca807ab7143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:54.831216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.831191    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a3caf3a-s6zmn\" (UID: \"f5212a51-1dab-44ae-9c46-2ca807ab7143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:54.833100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.833078    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a3caf3a-s6zmn\" (UID: \"f5212a51-1dab-44ae-9c46-2ca807ab7143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:54.838618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.838597    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qdqt\" (UniqueName: \"kubernetes.io/projected/f5212a51-1dab-44ae-9c46-2ca807ab7143-kube-api-access-8qdqt\") pod \"maas-api-revoke-sub-9a3caf3a-s6zmn\" (UID: \"f5212a51-1dab-44ae-9c46-2ca807ab7143\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:54.957021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:54.956940    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:55.037377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.037341    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:15:55.037557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.037447    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" podUID="5f719e78-068e-4c52-96c5-58eddc3de968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 21:15:55.262757 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:15:55.262725    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5212a51_1dab_44ae_9c46_2ca807ab7143.slice/crio-52e7e5e272edea78c5e0754a5ddde34217e9c244722dbbbe871f51ce1f0a6fc9 WatchSource:0}: Error finding container 52e7e5e272edea78c5e0754a5ddde34217e9c244722dbbbe871f51ce1f0a6fc9: Status 404 returned error can't find the container with id 52e7e5e272edea78c5e0754a5ddde34217e9c244722dbbbe871f51ce1f0a6fc9
Oct 30 21:15:55.263376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.263352    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"]
Oct 30 21:15:55.343306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.343283    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:55.443414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.443217    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-xks-ca\") pod \"a56a6e36-5652-409e-b2f0-e912f657c8c6\" (UID: \"a56a6e36-5652-409e-b2f0-e912f657c8c6\") "
Oct 30 21:15:55.443414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.443347    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a56a6e36-5652-409e-b2f0-e912f657c8c6-kube-api-access-t8wl9\" (UniqueName: \"kubernetes.io/projected/a56a6e36-5652-409e-b2f0-e912f657c8c6-kube-api-access-t8wl9\") pod \"a56a6e36-5652-409e-b2f0-e912f657c8c6\" (UID: \"a56a6e36-5652-409e-b2f0-e912f657c8c6\") "
Oct 30 21:15:55.443653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.443446    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-openshift-ca\") pod \"a56a6e36-5652-409e-b2f0-e912f657c8c6\" (UID: \"a56a6e36-5652-409e-b2f0-e912f657c8c6\") "
Oct 30 21:15:55.445156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.445082    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-openshift-ca" pod "a56a6e36-5652-409e-b2f0-e912f657c8c6" (UID: "a56a6e36-5652-409e-b2f0-e912f657c8c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:55.454308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.454257    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:55.517585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.517525    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-xks-ca" pod "a56a6e36-5652-409e-b2f0-e912f657c8c6" (UID: "a56a6e36-5652-409e-b2f0-e912f657c8c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:55.538706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.538662    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56a6e36-5652-409e-b2f0-e912f657c8c6-kube-api-access-t8wl9" pod "a56a6e36-5652-409e-b2f0-e912f657c8c6" (UID: "a56a6e36-5652-409e-b2f0-e912f657c8c6"). InnerVolumeSpecName "kube-api-access-t8wl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:55.555868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.555836    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a56a6e36-5652-409e-b2f0-e912f657c8c6-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:55.555868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:55.555863    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wl9\" (UniqueName: \"kubernetes.io/projected/a56a6e36-5652-409e-b2f0-e912f657c8c6-kube-api-access-t8wl9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:56.037040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:56.037006    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:15:56.037216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:56.037059    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" podUID="5f719e78-068e-4c52-96c5-58eddc3de968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 21:15:56.145579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:56.145491    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn" event={"ID":"f5212a51-1dab-44ae-9c46-2ca807ab7143","Type":"ContainerStarted","Data":"a4a06a0f44a5e76ad613d223f104b86eca5741b33da1526da484525db7cad601"}
Oct 30 21:15:56.145579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:56.145543    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn" event={"ID":"f5212a51-1dab-44ae-9c46-2ca807ab7143","Type":"ContainerStarted","Data":"52e7e5e272edea78c5e0754a5ddde34217e9c244722dbbbe871f51ce1f0a6fc9"}
Oct 30 21:15:56.147240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:56.147222    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"
Oct 30 21:15:56.151517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:56.147148    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn" event={"ID":"a56a6e36-5652-409e-b2f0-e912f657c8c6","Type":"ContainerDied","Data":"536e1175daa0ded30641dfeff7e493b8834924f981237112a780ff5af5ffce58"}
Oct 30 21:15:56.151596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:56.151546    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="536e1175daa0ded30641dfeff7e493b8834924f981237112a780ff5af5ffce58"
Oct 30 21:15:56.165740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:56.165697    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn" podStartSLOduration=2.165684347 podStartE2EDuration="2.165684347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:56.163021206 +0000 UTC m=+869.209580373" watchObservedRunningTime="2026-10-30 21:15:56.165684347 +0000 UTC m=+869.212243491"
Oct 30 21:15:57.037473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:57.037441    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 30 21:15:57.037855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:57.037502    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" podUID="5f719e78-068e-4c52-96c5-58eddc3de968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 30 21:15:57.233539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:57.233497    4918 generic.go:356] "Generic (PLEG): container finished" podID="f5212a51-1dab-44ae-9c46-2ca807ab7143" containerID="a4a06a0f44a5e76ad613d223f104b86eca5741b33da1526da484525db7cad601" exitCode=0
Oct 30 21:15:57.233729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:57.233709    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn" event={"ID":"f5212a51-1dab-44ae-9c46-2ca807ab7143","Type":"ContainerDied","Data":"a4a06a0f44a5e76ad613d223f104b86eca5741b33da1526da484525db7cad601"}
Oct 30 21:15:58.042018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.041988    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:58.042474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.042411    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:58.043626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.043605    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:15:58.389717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.389691    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:58.515855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.515827    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5212a51-1dab-44ae-9c46-2ca807ab7143-kube-api-access-8qdqt\" (UniqueName: \"kubernetes.io/projected/f5212a51-1dab-44ae-9c46-2ca807ab7143-kube-api-access-8qdqt\") pod \"f5212a51-1dab-44ae-9c46-2ca807ab7143\" (UID: \"f5212a51-1dab-44ae-9c46-2ca807ab7143\") "
Oct 30 21:15:58.516010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.515880    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-xks-ca\") pod \"f5212a51-1dab-44ae-9c46-2ca807ab7143\" (UID: \"f5212a51-1dab-44ae-9c46-2ca807ab7143\") "
Oct 30 21:15:58.516010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.515946    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-openshift-ca\") pod \"f5212a51-1dab-44ae-9c46-2ca807ab7143\" (UID: \"f5212a51-1dab-44ae-9c46-2ca807ab7143\") "
Oct 30 21:15:58.516341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.516316    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-openshift-ca" pod "f5212a51-1dab-44ae-9c46-2ca807ab7143" (UID: "f5212a51-1dab-44ae-9c46-2ca807ab7143"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:15:58.516604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.516587    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:58.518185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.518165    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-xks-ca" pod "f5212a51-1dab-44ae-9c46-2ca807ab7143" (UID: "f5212a51-1dab-44ae-9c46-2ca807ab7143"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:15:58.518269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.518255    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5212a51-1dab-44ae-9c46-2ca807ab7143-kube-api-access-8qdqt" pod "f5212a51-1dab-44ae-9c46-2ca807ab7143" (UID: "f5212a51-1dab-44ae-9c46-2ca807ab7143"). InnerVolumeSpecName "kube-api-access-8qdqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:15:58.617655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.617576    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qdqt\" (UniqueName: \"kubernetes.io/projected/f5212a51-1dab-44ae-9c46-2ca807ab7143-kube-api-access-8qdqt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:58.617655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:58.617602    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5212a51-1dab-44ae-9c46-2ca807ab7143-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:15:59.160059 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:15:59.160016    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:15:59.254448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:59.254409    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn" event={"ID":"f5212a51-1dab-44ae-9c46-2ca807ab7143","Type":"ContainerDied","Data":"52e7e5e272edea78c5e0754a5ddde34217e9c244722dbbbe871f51ce1f0a6fc9"}
Oct 30 21:15:59.254448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:59.254448    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"
Oct 30 21:15:59.254684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:15:59.254461    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52e7e5e272edea78c5e0754a5ddde34217e9c244722dbbbe871f51ce1f0a6fc9"
Oct 30 21:16:01.437417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:01.433490    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:01.615549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:01.615494    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:01.617278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:01.616993    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:16:01.690880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:01.690780    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:16:01.691237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:01.691109    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:01.716530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:01.716479    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:01.924598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:01.924555    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:01.934797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:01.934748    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:02.215418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.213264    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:02.435535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.435485    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:02.436624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.436600    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:16:02.607589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.607539    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:02.634040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.633979    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:02.781845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.637473    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:02.787437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.718050    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:02.791738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.718086    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"]
Oct 30 21:16:02.817416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.792735    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f5212a51-1dab-44ae-9c46-2ca807ab7143" containerName="revoke-keys"
Oct 30 21:16:02.817670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.817642    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5212a51-1dab-44ae-9c46-2ca807ab7143" containerName="revoke-keys"
Oct 30 21:16:02.817802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.817789    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a56a6e36-5652-409e-b2f0-e912f657c8c6" containerName="revoke-keys"
Oct 30 21:16:02.817908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.817895    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a56a6e36-5652-409e-b2f0-e912f657c8c6" containerName="revoke-keys"
Oct 30 21:16:02.818218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.818199    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5212a51-1dab-44ae-9c46-2ca807ab7143" containerName="revoke-keys"
Oct 30 21:16:02.818333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.818322    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a56a6e36-5652-409e-b2f0-e912f657c8c6" containerName="revoke-keys"
Oct 30 21:16:02.818532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.818497    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:02.818688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.818565    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" podUID="6521302f-c44e-40f0-b342-10c81833e27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:02.818688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.738512    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:16:02.818991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.738570    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:02.820448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.718107    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:16:02.882478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.871714    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:16:02.882478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.871776    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"]
Oct 30 21:16:02.882478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.871855    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:16:02.882478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.871911    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:16:02.882478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.871951    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:16:02.882478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.872010    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:16:02.882478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.878716    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:02.927418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.919588    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:02.984343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.984310    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:16:02.990416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:02.986646    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:16:03.025430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.017547    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:03.025430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.018663    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:16:03.025430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.020004    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:03.025430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.021224    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:16:03.055670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.055620    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnr4\" (UniqueName: \"kubernetes.io/projected/74a6aefd-c0c0-4db2-b058-c4560377140f-kube-api-access-fnnr4\") pod \"maas-api-revoke-sub-1626ea67-fl25x\" (UID: \"74a6aefd-c0c0-4db2-b058-c4560377140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:03.073838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.073802    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1626ea67-fl25x\" (UID: \"74a6aefd-c0c0-4db2-b058-c4560377140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:03.074208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.074188    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1626ea67-fl25x\" (UID: \"74a6aefd-c0c0-4db2-b058-c4560377140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:03.190632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.190536    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:16:03.219021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.218978    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:03.270885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.264301    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:16:03.347631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.279023    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnr4\" (UniqueName: \"kubernetes.io/projected/74a6aefd-c0c0-4db2-b058-c4560377140f-kube-api-access-fnnr4\") pod \"maas-api-revoke-sub-1626ea67-fl25x\" (UID: \"74a6aefd-c0c0-4db2-b058-c4560377140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:03.347631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.279131    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1626ea67-fl25x\" (UID: \"74a6aefd-c0c0-4db2-b058-c4560377140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:03.347631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.279205    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1626ea67-fl25x\" (UID: \"74a6aefd-c0c0-4db2-b058-c4560377140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:03.347631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.299592    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1626ea67-fl25x\" (UID: \"74a6aefd-c0c0-4db2-b058-c4560377140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:03.482572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.441151    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:03.482572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.459855    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:03.482572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.459928    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:03.482572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.460011    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:03.482572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.460035    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:03.630207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.630167    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:03.681418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.665525    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:16:03.797481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.782477    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:03.822493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.798279    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:03.829467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.816153    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:03.843979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.818780    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:03.845454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.819093    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1626ea67-fl25x\" (UID: \"74a6aefd-c0c0-4db2-b058-c4560377140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:03.845660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.819126    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:03.845967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.845443    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:16:03.875448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:03.846217    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:04.019417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.019090    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnr4\" (UniqueName: \"kubernetes.io/projected/74a6aefd-c0c0-4db2-b058-c4560377140f-kube-api-access-fnnr4\") pod \"maas-api-revoke-sub-1626ea67-fl25x\" (UID: \"74a6aefd-c0c0-4db2-b058-c4560377140f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:04.019417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.019191    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:04.044418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.021288    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:16:04.126349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.126029    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:04.284509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.241211    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:16:04.450414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.447173    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:04.636469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.635582    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:04.701417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.701190    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:04.821417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.783352    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:04.842617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.842571    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:04.847415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.843345    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:16:04.872830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.847709    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:16:04.896985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.847749    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:16:04.898840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:04.897229    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:16:05.059659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.059611    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:05.268311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.067668    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:05.280478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.264597    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:05.564477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.477536    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:16:05.713416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.696366    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:05.724685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.724619    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:05.807659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.807616    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:05.863953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.863878    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:16:05.864197 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.864184    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:16:05.864311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.864301    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:16:05.864486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.864472    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:16:05.864612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.864603    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:16:05.866728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.864329    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:05.870645 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.870071    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:16:05.871408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.871370    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:16:05.876822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.876807    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:16:05.877602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.877585    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:16:05.882913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.882896    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"]
Oct 30 21:16:05.890210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.890195    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"]
Oct 30 21:16:05.890467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.890455    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:16:05.891036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.890369    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:05.894808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.894793    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"]
Oct 30 21:16:05.896594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.896579    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:16:05.961459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.961429    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f386072-kgls5\" (UID: \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:05.961459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.961461    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcxvx\" (UniqueName: \"kubernetes.io/projected/d52459a4-e5af-4ba1-9237-3d4b6ef15540-kube-api-access-mcxvx\") pod \"maas-api-revoke-sub-0f386072-kgls5\" (UID: \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:05.961770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:05.961747    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f386072-kgls5\" (UID: \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:06.063970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:06.063938    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f386072-kgls5\" (UID: \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:06.063970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:06.063976    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcxvx\" (UniqueName: \"kubernetes.io/projected/d52459a4-e5af-4ba1-9237-3d4b6ef15540-kube-api-access-mcxvx\") pod \"maas-api-revoke-sub-0f386072-kgls5\" (UID: \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:06.064212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:06.064077    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f386072-kgls5\" (UID: \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:06.064719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:06.064691    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f386072-kgls5\" (UID: \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:06.086752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:06.086703    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f386072-kgls5\" (UID: \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:06.086912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:06.086884    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcxvx\" (UniqueName: \"kubernetes.io/projected/d52459a4-e5af-4ba1-9237-3d4b6ef15540-kube-api-access-mcxvx\") pod \"maas-api-revoke-sub-0f386072-kgls5\" (UID: \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:06.246906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:06.246875    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:06.531138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:06.531098    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"]
Oct 30 21:16:06.531358 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:16:06.531337    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52459a4_e5af_4ba1_9237_3d4b6ef15540.slice/crio-172662332f4e811b28dcd7b67fef04fa8e7e2d51c6e67616d2afca6275ea104a WatchSource:0}: Error finding container 172662332f4e811b28dcd7b67fef04fa8e7e2d51c6e67616d2afca6275ea104a: Status 404 returned error can't find the container with id 172662332f4e811b28dcd7b67fef04fa8e7e2d51c6e67616d2afca6275ea104a
Oct 30 21:16:06.808444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:06.808409    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x" event={"ID":"74a6aefd-c0c0-4db2-b058-c4560377140f","Type":"ContainerStarted","Data":"41c9879a7c7d21df4efd29c74b63be1484ff96e60858d431c92f2fe8eaa9c0f3"}
Oct 30 21:16:06.808811 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:06.808472    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x" event={"ID":"74a6aefd-c0c0-4db2-b058-c4560377140f","Type":"ContainerStarted","Data":"5574141d9ecfe6c244b0aff29e5942e199d569fbabd07a49f4b061202c8fc089"}
Oct 30 21:16:06.809866 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:06.809843    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5" event={"ID":"d52459a4-e5af-4ba1-9237-3d4b6ef15540","Type":"ContainerStarted","Data":"172662332f4e811b28dcd7b67fef04fa8e7e2d51c6e67616d2afca6275ea104a"}
Oct 30 21:16:07.818899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:07.818866    4918 generic.go:356] "Generic (PLEG): container finished" podID="74a6aefd-c0c0-4db2-b058-c4560377140f" containerID="41c9879a7c7d21df4efd29c74b63be1484ff96e60858d431c92f2fe8eaa9c0f3" exitCode=0
Oct 30 21:16:07.819328 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:07.818907    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x" event={"ID":"74a6aefd-c0c0-4db2-b058-c4560377140f","Type":"ContainerDied","Data":"41c9879a7c7d21df4efd29c74b63be1484ff96e60858d431c92f2fe8eaa9c0f3"}
Oct 30 21:16:07.820894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:07.820872    4918 generic.go:356] "Generic (PLEG): container finished" podID="d52459a4-e5af-4ba1-9237-3d4b6ef15540" containerID="cef1e6fa52a62d816845a950be48f257e80c252e8e408992cb7de6210c4f1e4f" exitCode=0
Oct 30 21:16:07.821019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:07.820950    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5" event={"ID":"d52459a4-e5af-4ba1-9237-3d4b6ef15540","Type":"ContainerDied","Data":"cef1e6fa52a62d816845a950be48f257e80c252e8e408992cb7de6210c4f1e4f"}
Oct 30 21:16:07.982483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:07.982463    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:08.087880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.087798    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a6aefd-c0c0-4db2-b058-c4560377140f-kube-api-access-fnnr4\" (UniqueName: \"kubernetes.io/projected/74a6aefd-c0c0-4db2-b058-c4560377140f-kube-api-access-fnnr4\") pod \"74a6aefd-c0c0-4db2-b058-c4560377140f\" (UID: \"74a6aefd-c0c0-4db2-b058-c4560377140f\") "
Oct 30 21:16:08.088051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.087945    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-xks-ca\") pod \"74a6aefd-c0c0-4db2-b058-c4560377140f\" (UID: \"74a6aefd-c0c0-4db2-b058-c4560377140f\") "
Oct 30 21:16:08.088051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.088007    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-openshift-ca\") pod \"74a6aefd-c0c0-4db2-b058-c4560377140f\" (UID: \"74a6aefd-c0c0-4db2-b058-c4560377140f\") "
Oct 30 21:16:08.088355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.088324    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-openshift-ca" pod "74a6aefd-c0c0-4db2-b058-c4560377140f" (UID: "74a6aefd-c0c0-4db2-b058-c4560377140f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:08.090081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.090055    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-xks-ca" pod "74a6aefd-c0c0-4db2-b058-c4560377140f" (UID: "74a6aefd-c0c0-4db2-b058-c4560377140f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:08.090186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.090154    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a6aefd-c0c0-4db2-b058-c4560377140f-kube-api-access-fnnr4" pod "74a6aefd-c0c0-4db2-b058-c4560377140f" (UID: "74a6aefd-c0c0-4db2-b058-c4560377140f"). InnerVolumeSpecName "kube-api-access-fnnr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:08.189523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.189487    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnnr4\" (UniqueName: \"kubernetes.io/projected/74a6aefd-c0c0-4db2-b058-c4560377140f-kube-api-access-fnnr4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.189523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.189517    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.189523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.189526    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a6aefd-c0c0-4db2-b058-c4560377140f-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:08.830531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.830498    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"
Oct 30 21:16:08.830908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.830527    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x" event={"ID":"74a6aefd-c0c0-4db2-b058-c4560377140f","Type":"ContainerDied","Data":"5574141d9ecfe6c244b0aff29e5942e199d569fbabd07a49f4b061202c8fc089"}
Oct 30 21:16:08.830908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.830571    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5574141d9ecfe6c244b0aff29e5942e199d569fbabd07a49f4b061202c8fc089"
Oct 30 21:16:08.962314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.962282    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"]
Oct 30 21:16:08.963147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.963128    4918 cpu_manager.go:414] "containerMap: removing container" podUID="74a6aefd-c0c0-4db2-b058-c4560377140f" containerName="revoke-keys"
Oct 30 21:16:08.963215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.963149    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a6aefd-c0c0-4db2-b058-c4560377140f" containerName="revoke-keys"
Oct 30 21:16:08.963320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.963308    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a6aefd-c0c0-4db2-b058-c4560377140f" containerName="revoke-keys"
Oct 30 21:16:08.969787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.969765    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:08.973158 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.973134    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"]
Oct 30 21:16:08.989906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:08.989879    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:09.108307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.108228    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52459a4-e5af-4ba1-9237-3d4b6ef15540-kube-api-access-mcxvx\" (UniqueName: \"kubernetes.io/projected/d52459a4-e5af-4ba1-9237-3d4b6ef15540-kube-api-access-mcxvx\") pod \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\" (UID: \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\") "
Oct 30 21:16:09.108503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.108322    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-openshift-ca\") pod \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\" (UID: \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\") "
Oct 30 21:16:09.108503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.108456    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-xks-ca\") pod \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\" (UID: \"d52459a4-e5af-4ba1-9237-3d4b6ef15540\") "
Oct 30 21:16:09.108687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.108664    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-openshift-ca" pod "d52459a4-e5af-4ba1-9237-3d4b6ef15540" (UID: "d52459a4-e5af-4ba1-9237-3d4b6ef15540"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:09.108813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.108795    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk99x\" (UniqueName: \"kubernetes.io/projected/f7501229-9b58-4af7-b99e-ae3d6f3ae599-kube-api-access-vk99x\") pod \"maas-api-revoke-sub-d2981703-q89x9\" (UID: \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:09.108883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.108863    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2981703-q89x9\" (UID: \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:09.109209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.109168    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2981703-q89x9\" (UID: \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:09.109320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.109291    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:09.110587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.110554    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52459a4-e5af-4ba1-9237-3d4b6ef15540-kube-api-access-mcxvx" pod "d52459a4-e5af-4ba1-9237-3d4b6ef15540" (UID: "d52459a4-e5af-4ba1-9237-3d4b6ef15540"). InnerVolumeSpecName "kube-api-access-mcxvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:09.110694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.110644    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-xks-ca" pod "d52459a4-e5af-4ba1-9237-3d4b6ef15540" (UID: "d52459a4-e5af-4ba1-9237-3d4b6ef15540"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:09.211050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.211013    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2981703-q89x9\" (UID: \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:09.211266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.211110    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk99x\" (UniqueName: \"kubernetes.io/projected/f7501229-9b58-4af7-b99e-ae3d6f3ae599-kube-api-access-vk99x\") pod \"maas-api-revoke-sub-d2981703-q89x9\" (UID: \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:09.211266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.211171    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2981703-q89x9\" (UID: \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:09.211266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.211240    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcxvx\" (UniqueName: \"kubernetes.io/projected/d52459a4-e5af-4ba1-9237-3d4b6ef15540-kube-api-access-mcxvx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:09.211266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.211257    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d52459a4-e5af-4ba1-9237-3d4b6ef15540-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:09.211838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.211819    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2981703-q89x9\" (UID: \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:09.213556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.213537    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2981703-q89x9\" (UID: \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:09.219520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.219495    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk99x\" (UniqueName: \"kubernetes.io/projected/f7501229-9b58-4af7-b99e-ae3d6f3ae599-kube-api-access-vk99x\") pod \"maas-api-revoke-sub-d2981703-q89x9\" (UID: \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:09.300574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.300541    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:09.582566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.582537    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"]
Oct 30 21:16:09.582939 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:16:09.582912    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7501229_9b58_4af7_b99e_ae3d6f3ae599.slice/crio-50cf0f69d344aa3617d99b5d3f971bfa5c685b548d49eb69347ca9358d0484e1 WatchSource:0}: Error finding container 50cf0f69d344aa3617d99b5d3f971bfa5c685b548d49eb69347ca9358d0484e1: Status 404 returned error can't find the container with id 50cf0f69d344aa3617d99b5d3f971bfa5c685b548d49eb69347ca9358d0484e1
Oct 30 21:16:09.838773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.838738    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5" event={"ID":"d52459a4-e5af-4ba1-9237-3d4b6ef15540","Type":"ContainerDied","Data":"172662332f4e811b28dcd7b67fef04fa8e7e2d51c6e67616d2afca6275ea104a"}
Oct 30 21:16:09.839162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.838818    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="172662332f4e811b28dcd7b67fef04fa8e7e2d51c6e67616d2afca6275ea104a"
Oct 30 21:16:09.839162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.838749    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"
Oct 30 21:16:09.840067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:09.840036    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9" event={"ID":"f7501229-9b58-4af7-b99e-ae3d6f3ae599","Type":"ContainerStarted","Data":"50cf0f69d344aa3617d99b5d3f971bfa5c685b548d49eb69347ca9358d0484e1"}
Oct 30 21:16:10.849312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:10.849276    4918 generic.go:356] "Generic (PLEG): container finished" podID="f7501229-9b58-4af7-b99e-ae3d6f3ae599" containerID="0e39e4a22335917c7248b9e056e586daccf240da131887cab0c7b6789e4e619e" exitCode=0
Oct 30 21:16:10.849734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:10.849365    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9" event={"ID":"f7501229-9b58-4af7-b99e-ae3d6f3ae599","Type":"ContainerDied","Data":"0e39e4a22335917c7248b9e056e586daccf240da131887cab0c7b6789e4e619e"}
Oct 30 21:16:12.818421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:12.809751    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:16:12.862467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:12.853494    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:12.862467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:12.854057    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:12.973417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:12.934798    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:13.036416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:12.974505    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:16:13.036416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:12.976323    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:13.036416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:12.976807    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:13.036416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.013481    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:16:13.430920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.430872    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:13.516424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.509014    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:13.776605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.776575    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:13.808596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.808562    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-openshift-ca\") pod \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\" (UID: \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\") "
Oct 30 21:16:13.809299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.809275    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-xks-ca\") pod \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\" (UID: \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\") "
Oct 30 21:16:13.809960 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.809939    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7501229-9b58-4af7-b99e-ae3d6f3ae599-kube-api-access-vk99x\" (UniqueName: \"kubernetes.io/projected/f7501229-9b58-4af7-b99e-ae3d6f3ae599-kube-api-access-vk99x\") pod \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\" (UID: \"f7501229-9b58-4af7-b99e-ae3d6f3ae599\") "
Oct 30 21:16:13.810895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.809167    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-openshift-ca" pod "f7501229-9b58-4af7-b99e-ae3d6f3ae599" (UID: "f7501229-9b58-4af7-b99e-ae3d6f3ae599"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:13.828508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.828467    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:13.829788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.829767    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:16:13.884417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.854102    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:13.884417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.854753    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:16:13.884417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.856011    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:16:13.884417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.856293    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:16:13.932428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.912232    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:13.975422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.974224    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:13.975422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.974625    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:16:14.000432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.978628    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:14.000432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.978924    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:16:14.000432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.999022    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:14.000432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.999407    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:16:14.000432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.999721    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:16:14.000432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:13.999779    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:16:14.025327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.025285    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:14.025937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.025594    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:14.026921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.026863    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:16:14.078692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.078571    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7501229-9b58-4af7-b99e-ae3d6f3ae599-kube-api-access-vk99x" pod "f7501229-9b58-4af7-b99e-ae3d6f3ae599" (UID: "f7501229-9b58-4af7-b99e-ae3d6f3ae599"). InnerVolumeSpecName "kube-api-access-vk99x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:14.123368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.123298    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-xks-ca" pod "f7501229-9b58-4af7-b99e-ae3d6f3ae599" (UID: "f7501229-9b58-4af7-b99e-ae3d6f3ae599"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:14.174426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.165223    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7501229-9b58-4af7-b99e-ae3d6f3ae599-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:14.174683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.174659    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk99x\" (UniqueName: \"kubernetes.io/projected/f7501229-9b58-4af7-b99e-ae3d6f3ae599-kube-api-access-vk99x\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:14.431473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.430076    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:14.431473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.430136    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" podUID="5f719e78-068e-4c52-96c5-58eddc3de968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:14.463414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.444513    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:14.463414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.445265    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:16:14.650086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.613771    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:14.653419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.652887    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:16:14.653419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.652945    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:16:14.653419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.652979    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:16:14.654519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.654490    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:16:14.667434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.656044    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:16:14.695420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.685476    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9" event={"ID":"f7501229-9b58-4af7-b99e-ae3d6f3ae599","Type":"ContainerDied","Data":"50cf0f69d344aa3617d99b5d3f971bfa5c685b548d49eb69347ca9358d0484e1"}
Oct 30 21:16:14.695420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.685545    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50cf0f69d344aa3617d99b5d3f971bfa5c685b548d49eb69347ca9358d0484e1"
Oct 30 21:16:14.695420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.685617    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"
Oct 30 21:16:14.717528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.717487    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:14.843131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.842516    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:16:14.887922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.886340    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:16:14.887922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.886434    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:16:14.915878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.915848    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:14.938519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:14.938487    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" podUID="ae65e4e9-386b-440c-b568-2402f61346b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:15.001560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.001520    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:15.036083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.036046    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:15.064420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.062958    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:16:15.373553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.373469    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:16:15.373827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.373804    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:16:15.379424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.379374    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:16:15.380181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.380161    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:16:15.400886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.400857    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:16:15.410558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.410520    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"]
Oct 30 21:16:15.413776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.413749    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d52459a4-e5af-4ba1-9237-3d4b6ef15540" containerName="revoke-keys"
Oct 30 21:16:15.413977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.413961    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52459a4-e5af-4ba1-9237-3d4b6ef15540" containerName="revoke-keys"
Oct 30 21:16:15.414096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.414086    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f7501229-9b58-4af7-b99e-ae3d6f3ae599" containerName="revoke-keys"
Oct 30 21:16:15.414163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.414153    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7501229-9b58-4af7-b99e-ae3d6f3ae599" containerName="revoke-keys"
Oct 30 21:16:15.414427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.414408    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52459a4-e5af-4ba1-9237-3d4b6ef15540" containerName="revoke-keys"
Oct 30 21:16:15.416567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.416550    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7501229-9b58-4af7-b99e-ae3d6f3ae599" containerName="revoke-keys"
Oct 30 21:16:15.445445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.439729    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.490820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.489318    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.491538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.491372    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.493683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.493656    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.496315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.495299    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.532470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.532436    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.538249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.537585    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0dc22c-2bed-4082-9a17-c606d796ce30-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.538538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.538500    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.538809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.538786    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v7j5\" (UniqueName: \"kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-kube-api-access-5v7j5\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.539081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.539016    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.587022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.586987    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"]
Oct 30 21:16:15.641033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.640949    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.641033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.640990    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0dc22c-2bed-4082-9a17-c606d796ce30-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.641033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.641021    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.641251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.641102    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v7j5\" (UniqueName: \"kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-kube-api-access-5v7j5\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.641251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.641142    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.641251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.641195    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.641251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.641225    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.641376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.641276    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.641376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.641330    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.641751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.641725    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.641975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.641792    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.641975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.641894    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0dc22c-2bed-4082-9a17-c606d796ce30-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.642277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.642236    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.642485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.642464    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.644153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.644134    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.644421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.644378    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.651169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.651145    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.651498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.651481    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v7j5\" (UniqueName: \"kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-kube-api-access-5v7j5\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:15.844466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:15.844432    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:16.131711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:16.131684    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"]
Oct 30 21:16:16.132200 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:16:16.132182    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0dc22c_2bed_4082_9a17_c606d796ce30.slice/crio-0284556302ed4d3fb59f8fa4f7450862969cb5055f94916b3b4f850c6dd78c5c WatchSource:0}: Error finding container 0284556302ed4d3fb59f8fa4f7450862969cb5055f94916b3b4f850c6dd78c5c: Status 404 returned error can't find the container with id 0284556302ed4d3fb59f8fa4f7450862969cb5055f94916b3b4f850c6dd78c5c
Oct 30 21:16:16.134682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:16.134654    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:16.134754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:16.134718    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:16.134754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:16.134744    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:16.718772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:16.718732    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv" event={"ID":"6d0dc22c-2bed-4082-9a17-c606d796ce30","Type":"ContainerStarted","Data":"99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379"}
Oct 30 21:16:16.718976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:16.718802    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv" event={"ID":"6d0dc22c-2bed-4082-9a17-c606d796ce30","Type":"ContainerStarted","Data":"0284556302ed4d3fb59f8fa4f7450862969cb5055f94916b3b4f850c6dd78c5c"}
Oct 30 21:16:16.745589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:16.745543    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv" podStartSLOduration=2.745529234 podStartE2EDuration="2.745529234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:16.740686636 +0000 UTC m=+889.787245825" watchObservedRunningTime="2026-10-30 21:16:16.745529234 +0000 UTC m=+889.792088380"
Oct 30 21:16:16.845252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:16.845222    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:17.845933 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:17.845901    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 21:16:17.846327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:17.845955    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv" podUID="6d0dc22c-2bed-4082-9a17-c606d796ce30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 21:16:17.918837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:17.918806    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"]
Oct 30 21:16:17.923173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:17.923154    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:17.933352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:17.933324    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:17.934041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:17.934022    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"]
Oct 30 21:16:17.934998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:17.934976    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:16:17.934998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:17.934980    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:16:17.935486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:17.935466    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:16:18.097679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.097584    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72tbm\" (UniqueName: \"kubernetes.io/projected/74bd00e5-5f17-467b-952a-0f7ee99258c8-kube-api-access-72tbm\") pod \"maas-api-revoke-sub-31570acd-5cf7w\" (UID: \"74bd00e5-5f17-467b-952a-0f7ee99258c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:18.097922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.097902    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31570acd-5cf7w\" (UID: \"74bd00e5-5f17-467b-952a-0f7ee99258c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:18.098146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.098128    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31570acd-5cf7w\" (UID: \"74bd00e5-5f17-467b-952a-0f7ee99258c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:18.210306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.210261    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72tbm\" (UniqueName: \"kubernetes.io/projected/74bd00e5-5f17-467b-952a-0f7ee99258c8-kube-api-access-72tbm\") pod \"maas-api-revoke-sub-31570acd-5cf7w\" (UID: \"74bd00e5-5f17-467b-952a-0f7ee99258c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:18.210611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.210581    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31570acd-5cf7w\" (UID: \"74bd00e5-5f17-467b-952a-0f7ee99258c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:18.210751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.210722    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31570acd-5cf7w\" (UID: \"74bd00e5-5f17-467b-952a-0f7ee99258c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:18.211532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.211508    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31570acd-5cf7w\" (UID: \"74bd00e5-5f17-467b-952a-0f7ee99258c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:18.213199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.213176    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31570acd-5cf7w\" (UID: \"74bd00e5-5f17-467b-952a-0f7ee99258c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:18.219422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.219383    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72tbm\" (UniqueName: \"kubernetes.io/projected/74bd00e5-5f17-467b-952a-0f7ee99258c8-kube-api-access-72tbm\") pod \"maas-api-revoke-sub-31570acd-5cf7w\" (UID: \"74bd00e5-5f17-467b-952a-0f7ee99258c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:18.238790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.238767    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:18.564038 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:16:18.564002    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bd00e5_5f17_467b_952a_0f7ee99258c8.slice/crio-ae3a9c78d06feb58f24e3f70b2ac52e591beeda004db42e92ebca7bd6bc643a1 WatchSource:0}: Error finding container ae3a9c78d06feb58f24e3f70b2ac52e591beeda004db42e92ebca7bd6bc643a1: Status 404 returned error can't find the container with id ae3a9c78d06feb58f24e3f70b2ac52e591beeda004db42e92ebca7bd6bc643a1
Oct 30 21:16:18.565268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.565241    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"]
Oct 30 21:16:18.735422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.735356    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w" event={"ID":"74bd00e5-5f17-467b-952a-0f7ee99258c8","Type":"ContainerStarted","Data":"ae3a9c78d06feb58f24e3f70b2ac52e591beeda004db42e92ebca7bd6bc643a1"}
Oct 30 21:16:18.844852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.844827    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 21:16:18.844985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:18.844876    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv" podUID="6d0dc22c-2bed-4082-9a17-c606d796ce30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 21:16:19.228466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.219214    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"]
Oct 30 21:16:19.242598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.242569    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"]
Oct 30 21:16:19.242805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.242737    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:19.434259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.434223    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0897c0d-f8jqg\" (UID: \"36945330-cd5e-4c05-94e4-5304ac127bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:19.434471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.434314    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z7q5\" (UniqueName: \"kubernetes.io/projected/36945330-cd5e-4c05-94e4-5304ac127bb4-kube-api-access-9z7q5\") pod \"maas-api-revoke-sub-d0897c0d-f8jqg\" (UID: \"36945330-cd5e-4c05-94e4-5304ac127bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:19.434471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.434385    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0897c0d-f8jqg\" (UID: \"36945330-cd5e-4c05-94e4-5304ac127bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:19.535810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.535726    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z7q5\" (UniqueName: \"kubernetes.io/projected/36945330-cd5e-4c05-94e4-5304ac127bb4-kube-api-access-9z7q5\") pod \"maas-api-revoke-sub-d0897c0d-f8jqg\" (UID: \"36945330-cd5e-4c05-94e4-5304ac127bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:19.535810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.535779    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0897c0d-f8jqg\" (UID: \"36945330-cd5e-4c05-94e4-5304ac127bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:19.536008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.535925    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0897c0d-f8jqg\" (UID: \"36945330-cd5e-4c05-94e4-5304ac127bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:19.536671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.536575    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0897c0d-f8jqg\" (UID: \"36945330-cd5e-4c05-94e4-5304ac127bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:19.538381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.538358    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0897c0d-f8jqg\" (UID: \"36945330-cd5e-4c05-94e4-5304ac127bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:19.544184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.544162    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z7q5\" (UniqueName: \"kubernetes.io/projected/36945330-cd5e-4c05-94e4-5304ac127bb4-kube-api-access-9z7q5\") pod \"maas-api-revoke-sub-d0897c0d-f8jqg\" (UID: \"36945330-cd5e-4c05-94e4-5304ac127bb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:19.564131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.564104    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:19.745429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.745105    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w" event={"ID":"74bd00e5-5f17-467b-952a-0f7ee99258c8","Type":"ContainerStarted","Data":"f8de87e64b012ae42a2740c5a99d44f7f69453d49545187bd706cb27a6efac9d"}
Oct 30 21:16:19.772968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.772923    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w" podStartSLOduration=2.772909824 podStartE2EDuration="2.772909824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:19.771770467 +0000 UTC m=+892.818329609" watchObservedRunningTime="2026-10-30 21:16:19.772909824 +0000 UTC m=+892.819468969"
Oct 30 21:16:19.845461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.845431    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 21:16:19.845590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.845489    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv" podUID="6d0dc22c-2bed-4082-9a17-c606d796ce30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 21:16:19.883740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:19.883715    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"]
Oct 30 21:16:19.912549 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:16:19.912513    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36945330_cd5e_4c05_94e4_5304ac127bb4.slice/crio-f146833dfe3e486bc66de7716435195f4627f9158ae713d0cd0a1ab874b3becf WatchSource:0}: Error finding container f146833dfe3e486bc66de7716435195f4627f9158ae713d0cd0a1ab874b3becf: Status 404 returned error can't find the container with id f146833dfe3e486bc66de7716435195f4627f9158ae713d0cd0a1ab874b3becf
Oct 30 21:16:20.754434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:20.754381    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg" event={"ID":"36945330-cd5e-4c05-94e4-5304ac127bb4","Type":"ContainerStarted","Data":"1b09c3baf6c3d334bf3266e8f055709eec5b3440a9f62f4bed9822fbd6f57f43"}
Oct 30 21:16:20.754927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:20.754467    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg" event={"ID":"36945330-cd5e-4c05-94e4-5304ac127bb4","Type":"ContainerStarted","Data":"f146833dfe3e486bc66de7716435195f4627f9158ae713d0cd0a1ab874b3becf"}
Oct 30 21:16:20.756563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:20.756538    4918 generic.go:356] "Generic (PLEG): container finished" podID="74bd00e5-5f17-467b-952a-0f7ee99258c8" containerID="f8de87e64b012ae42a2740c5a99d44f7f69453d49545187bd706cb27a6efac9d" exitCode=0
Oct 30 21:16:20.756707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:20.756630    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w" event={"ID":"74bd00e5-5f17-467b-952a-0f7ee99258c8","Type":"ContainerDied","Data":"f8de87e64b012ae42a2740c5a99d44f7f69453d49545187bd706cb27a6efac9d"}
Oct 30 21:16:20.814919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:20.814856    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg" podStartSLOduration=1.814839724 podStartE2EDuration="1.814839724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:20.812757049 +0000 UTC m=+893.859316221" watchObservedRunningTime="2026-10-30 21:16:20.814839724 +0000 UTC m=+893.861398869"
Oct 30 21:16:20.845131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:20.845102    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 21:16:20.845260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:20.845151    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv" podUID="6d0dc22c-2bed-4082-9a17-c606d796ce30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 21:16:21.765710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:21.765678    4918 generic.go:356] "Generic (PLEG): container finished" podID="36945330-cd5e-4c05-94e4-5304ac127bb4" containerID="1b09c3baf6c3d334bf3266e8f055709eec5b3440a9f62f4bed9822fbd6f57f43" exitCode=0
Oct 30 21:16:21.766217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:21.765793    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg" event={"ID":"36945330-cd5e-4c05-94e4-5304ac127bb4","Type":"ContainerDied","Data":"1b09c3baf6c3d334bf3266e8f055709eec5b3440a9f62f4bed9822fbd6f57f43"}
Oct 30 21:16:21.846577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:21.845477    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 21:16:21.846577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:21.845521    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv" podUID="6d0dc22c-2bed-4082-9a17-c606d796ce30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 21:16:21.957308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:21.957283    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:22.024219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.024131    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-xks-ca\") pod \"74bd00e5-5f17-467b-952a-0f7ee99258c8\" (UID: \"74bd00e5-5f17-467b-952a-0f7ee99258c8\") "
Oct 30 21:16:22.024219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.024188    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-openshift-ca\") pod \"74bd00e5-5f17-467b-952a-0f7ee99258c8\" (UID: \"74bd00e5-5f17-467b-952a-0f7ee99258c8\") "
Oct 30 21:16:22.024436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.024420    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74bd00e5-5f17-467b-952a-0f7ee99258c8-kube-api-access-72tbm\" (UniqueName: \"kubernetes.io/projected/74bd00e5-5f17-467b-952a-0f7ee99258c8-kube-api-access-72tbm\") pod \"74bd00e5-5f17-467b-952a-0f7ee99258c8\" (UID: \"74bd00e5-5f17-467b-952a-0f7ee99258c8\") "
Oct 30 21:16:22.024562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.024538    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-openshift-ca" pod "74bd00e5-5f17-467b-952a-0f7ee99258c8" (UID: "74bd00e5-5f17-467b-952a-0f7ee99258c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:22.024974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.024955    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:22.026501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.026460    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-xks-ca" pod "74bd00e5-5f17-467b-952a-0f7ee99258c8" (UID: "74bd00e5-5f17-467b-952a-0f7ee99258c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:22.026838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.026818    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bd00e5-5f17-467b-952a-0f7ee99258c8-kube-api-access-72tbm" pod "74bd00e5-5f17-467b-952a-0f7ee99258c8" (UID: "74bd00e5-5f17-467b-952a-0f7ee99258c8"). InnerVolumeSpecName "kube-api-access-72tbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:22.126156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.126127    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72tbm\" (UniqueName: \"kubernetes.io/projected/74bd00e5-5f17-467b-952a-0f7ee99258c8-kube-api-access-72tbm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:22.126156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.126155    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74bd00e5-5f17-467b-952a-0f7ee99258c8-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:22.773578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.773547    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"
Oct 30 21:16:22.773578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.773570    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w" event={"ID":"74bd00e5-5f17-467b-952a-0f7ee99258c8","Type":"ContainerDied","Data":"ae3a9c78d06feb58f24e3f70b2ac52e591beeda004db42e92ebca7bd6bc643a1"}
Oct 30 21:16:22.774014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.773644    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae3a9c78d06feb58f24e3f70b2ac52e591beeda004db42e92ebca7bd6bc643a1"
Oct 30 21:16:22.847039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.845713    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 30 21:16:22.847039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.845767    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv" podUID="6d0dc22c-2bed-4082-9a17-c606d796ce30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 30 21:16:22.951907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:22.951883    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:23.041924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.041841    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-xks-ca\") pod \"36945330-cd5e-4c05-94e4-5304ac127bb4\" (UID: \"36945330-cd5e-4c05-94e4-5304ac127bb4\") "
Oct 30 21:16:23.041924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.041896    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-openshift-ca\") pod \"36945330-cd5e-4c05-94e4-5304ac127bb4\" (UID: \"36945330-cd5e-4c05-94e4-5304ac127bb4\") "
Oct 30 21:16:23.042112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.042031    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36945330-cd5e-4c05-94e4-5304ac127bb4-kube-api-access-9z7q5\" (UniqueName: \"kubernetes.io/projected/36945330-cd5e-4c05-94e4-5304ac127bb4-kube-api-access-9z7q5\") pod \"36945330-cd5e-4c05-94e4-5304ac127bb4\" (UID: \"36945330-cd5e-4c05-94e4-5304ac127bb4\") "
Oct 30 21:16:23.042444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.042413    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-openshift-ca" pod "36945330-cd5e-4c05-94e4-5304ac127bb4" (UID: "36945330-cd5e-4c05-94e4-5304ac127bb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:23.042846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.042820    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:23.044250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.044226    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-xks-ca" pod "36945330-cd5e-4c05-94e4-5304ac127bb4" (UID: "36945330-cd5e-4c05-94e4-5304ac127bb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:23.044739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.044718    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36945330-cd5e-4c05-94e4-5304ac127bb4-kube-api-access-9z7q5" pod "36945330-cd5e-4c05-94e4-5304ac127bb4" (UID: "36945330-cd5e-4c05-94e4-5304ac127bb4"). InnerVolumeSpecName "kube-api-access-9z7q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:23.158698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.158051    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36945330-cd5e-4c05-94e4-5304ac127bb4-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:23.158698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.158090    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z7q5\" (UniqueName: \"kubernetes.io/projected/36945330-cd5e-4c05-94e4-5304ac127bb4-kube-api-access-9z7q5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:23.473145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.472652    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"]
Oct 30 21:16:23.473480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.473458    4918 cpu_manager.go:414] "containerMap: removing container" podUID="36945330-cd5e-4c05-94e4-5304ac127bb4" containerName="revoke-keys"
Oct 30 21:16:23.473598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.473481    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36945330-cd5e-4c05-94e4-5304ac127bb4" containerName="revoke-keys"
Oct 30 21:16:23.473598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.473497    4918 cpu_manager.go:414] "containerMap: removing container" podUID="74bd00e5-5f17-467b-952a-0f7ee99258c8" containerName="revoke-keys"
Oct 30 21:16:23.473598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.473505    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74bd00e5-5f17-467b-952a-0f7ee99258c8" containerName="revoke-keys"
Oct 30 21:16:23.473751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.473647    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="74bd00e5-5f17-467b-952a-0f7ee99258c8" containerName="revoke-keys"
Oct 30 21:16:23.473751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.473667    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="36945330-cd5e-4c05-94e4-5304ac127bb4" containerName="revoke-keys"
Oct 30 21:16:23.501626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.501603    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:23.502174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.502151    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"]
Oct 30 21:16:23.569720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.569072    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwv7w\" (UniqueName: \"kubernetes.io/projected/4b9d8f7e-082b-438a-b44f-5020dfdd616a-kube-api-access-zwv7w\") pod \"maas-api-revoke-sub-a019c653-qgggf\" (UID: \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:23.569720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.569252    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a019c653-qgggf\" (UID: \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:23.575338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.575286    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a019c653-qgggf\" (UID: \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:23.676160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.676127    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwv7w\" (UniqueName: \"kubernetes.io/projected/4b9d8f7e-082b-438a-b44f-5020dfdd616a-kube-api-access-zwv7w\") pod \"maas-api-revoke-sub-a019c653-qgggf\" (UID: \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:23.676327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.676255    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a019c653-qgggf\" (UID: \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:23.676327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.676298    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a019c653-qgggf\" (UID: \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:23.677177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.677145    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a019c653-qgggf\" (UID: \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:23.679003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.678983    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a019c653-qgggf\" (UID: \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:23.687974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.687954    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwv7w\" (UniqueName: \"kubernetes.io/projected/4b9d8f7e-082b-438a-b44f-5020dfdd616a-kube-api-access-zwv7w\") pod \"maas-api-revoke-sub-a019c653-qgggf\" (UID: \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:23.783106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.783029    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"
Oct 30 21:16:23.783106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.783034    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg" event={"ID":"36945330-cd5e-4c05-94e4-5304ac127bb4","Type":"ContainerDied","Data":"f146833dfe3e486bc66de7716435195f4627f9158ae713d0cd0a1ab874b3becf"}
Oct 30 21:16:23.783106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.783082    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f146833dfe3e486bc66de7716435195f4627f9158ae713d0cd0a1ab874b3becf"
Oct 30 21:16:23.816814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.816791    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:23.863747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.863713    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:23.872717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.865349    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:23.892420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:23.883230    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:16:24.213127 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:16:24.213095    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9d8f7e_082b_438a_b44f_5020dfdd616a.slice/crio-5c57b2649426882afbb7d791fd3aa780dc5d9700222706f861de9f858f94ef6b WatchSource:0}: Error finding container 5c57b2649426882afbb7d791fd3aa780dc5d9700222706f861de9f858f94ef6b: Status 404 returned error can't find the container with id 5c57b2649426882afbb7d791fd3aa780dc5d9700222706f861de9f858f94ef6b
Oct 30 21:16:24.213647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:24.213623    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"]
Oct 30 21:16:24.793666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:24.793583    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf" event={"ID":"4b9d8f7e-082b-438a-b44f-5020dfdd616a","Type":"ContainerStarted","Data":"7fd7338bbc89f1a4813d6cc12e12fef08a947397236f6c7c16af1b9b1aae86b9"}
Oct 30 21:16:24.793666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:24.793640    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf" event={"ID":"4b9d8f7e-082b-438a-b44f-5020dfdd616a","Type":"ContainerStarted","Data":"5c57b2649426882afbb7d791fd3aa780dc5d9700222706f861de9f858f94ef6b"}
Oct 30 21:16:24.810890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:24.810843    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf" podStartSLOduration=1.810830214 podStartE2EDuration="1.810830214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:24.810139912 +0000 UTC m=+897.856699101" watchObservedRunningTime="2026-10-30 21:16:24.810830214 +0000 UTC m=+897.857389359"
Oct 30 21:16:25.903418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:25.899658    4918 generic.go:356] "Generic (PLEG): container finished" podID="4b9d8f7e-082b-438a-b44f-5020dfdd616a" containerID="7fd7338bbc89f1a4813d6cc12e12fef08a947397236f6c7c16af1b9b1aae86b9" exitCode=0
Oct 30 21:16:25.903418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:25.901440    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf" event={"ID":"4b9d8f7e-082b-438a-b44f-5020dfdd616a","Type":"ContainerDied","Data":"7fd7338bbc89f1a4813d6cc12e12fef08a947397236f6c7c16af1b9b1aae86b9"}
Oct 30 21:16:26.258416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:26.255570    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:26.913414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:26.912473    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:26.915423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:26.914205    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:16:27.375418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:27.355747    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:27.400417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:27.399205    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:27.488416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:27.436379    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:16:27.488416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:27.484637    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:16:27.709468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:27.685162    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:27.709468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:27.685724    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:16:27.709468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:27.693967    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:27.709468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:27.694049    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:16:27.709468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:27.707037    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:16:28.116422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.097445    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:28.493382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.152650    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:16:28.504829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.166514    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:16:28.504829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.502523    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:16:28.504829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.502583    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:16:28.504829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.445770    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:28.504829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.445880    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:16:28.504829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.448164    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:28.504829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.504088    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:16:28.504829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.504129    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:16:28.557736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.557686    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:28.558382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.558361    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:16:28.717568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.717520    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:16:28.718106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.718068    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:16:28.718505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.716881    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:28.765553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.720083    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:16:28.765553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.760302    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:16:28.765553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.760344    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:16:28.796839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.794002    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:28.796839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.794311    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:16:28.939568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.939539    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:16:28.995081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:28.994449    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:16:29.122463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.094253    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:16:29.501804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.498710    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": context deadline exceeded"
Oct 30 21:16:29.501804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.500291    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"]
Oct 30 21:16:29.537003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.536908    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:29.537227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.537202    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:16:29.537665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.537558    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:16:29.566447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.563651    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:29.607951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.607920    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:16:29.608210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.608194    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:16:29.608343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.608331    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:16:29.608573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.608556    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:29.641785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.641759    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:16:29.689166 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.665126    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:16:29.718615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.718589    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:16:29.732794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.717235    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncr8w\" (UniqueName: \"kubernetes.io/projected/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-kube-api-access-ncr8w\") pod \"maas-api-revoke-sub-5da633f5-trnrj\" (UID: \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:29.742064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.733106    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5da633f5-trnrj\" (UID: \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:29.808492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.808458    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5da633f5-trnrj\" (UID: \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:29.811168 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:16:29.796021    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:16:29.811486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.731022    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:29.811709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.810624    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"]
Oct 30 21:16:29.812515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.812498    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:16:29.818774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.818755    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:16:29.824605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.824588    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:16:29.922264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.915636    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-xks-ca\") pod \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\" (UID: \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\") "
Oct 30 21:16:29.923241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.923216    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-openshift-ca\") pod \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\" (UID: \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\") "
Oct 30 21:16:29.923827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.923803    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9d8f7e-082b-438a-b44f-5020dfdd616a-kube-api-access-zwv7w\" (UniqueName: \"kubernetes.io/projected/4b9d8f7e-082b-438a-b44f-5020dfdd616a-kube-api-access-zwv7w\") pod \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\" (UID: \"4b9d8f7e-082b-438a-b44f-5020dfdd616a\") "
Oct 30 21:16:29.927158 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.927122    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5da633f5-trnrj\" (UID: \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:29.929612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.929575    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5da633f5-trnrj\" (UID: \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:29.929856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.929835    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncr8w\" (UniqueName: \"kubernetes.io/projected/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-kube-api-access-ncr8w\") pod \"maas-api-revoke-sub-5da633f5-trnrj\" (UID: \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:29.934321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.930534    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5da633f5-trnrj\" (UID: \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:29.934515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.934477    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-openshift-ca" pod "4b9d8f7e-082b-438a-b44f-5020dfdd616a" (UID: "4b9d8f7e-082b-438a-b44f-5020dfdd616a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:29.936405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.936364    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"
Oct 30 21:16:29.937599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.937573    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf" event={"ID":"4b9d8f7e-082b-438a-b44f-5020dfdd616a","Type":"ContainerDied","Data":"5c57b2649426882afbb7d791fd3aa780dc5d9700222706f861de9f858f94ef6b"}
Oct 30 21:16:29.937694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.937658    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c57b2649426882afbb7d791fd3aa780dc5d9700222706f861de9f858f94ef6b"
Oct 30 21:16:29.944840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.944337    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5da633f5-trnrj\" (UID: \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:29.954085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.954036    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-xks-ca" pod "4b9d8f7e-082b-438a-b44f-5020dfdd616a" (UID: "4b9d8f7e-082b-438a-b44f-5020dfdd616a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:29.957458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.957181    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncr8w\" (UniqueName: \"kubernetes.io/projected/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-kube-api-access-ncr8w\") pod \"maas-api-revoke-sub-5da633f5-trnrj\" (UID: \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:29.958059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.958039    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9d8f7e-082b-438a-b44f-5020dfdd616a-kube-api-access-zwv7w" pod "4b9d8f7e-082b-438a-b44f-5020dfdd616a" (UID: "4b9d8f7e-082b-438a-b44f-5020dfdd616a"). InnerVolumeSpecName "kube-api-access-zwv7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:29.976809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.976785    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:16:29.980322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:29.980297    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:16:30.003474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.003447    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:30.043435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.042602    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwv7w\" (UniqueName: \"kubernetes.io/projected/4b9d8f7e-082b-438a-b44f-5020dfdd616a-kube-api-access-zwv7w\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:30.043435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.042624    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:30.043435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.042638    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b9d8f7e-082b-438a-b44f-5020dfdd616a-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:30.305667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.305637    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"]
Oct 30 21:16:30.306018 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:16:30.305704    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc720ed_be58_4f75_a5d2_5599b7ab7f4f.slice/crio-edda3768eeeaa0dcd809771dbf0cda2a67d4a9adae1f988ee4df601164dc39db WatchSource:0}: Error finding container edda3768eeeaa0dcd809771dbf0cda2a67d4a9adae1f988ee4df601164dc39db: Status 404 returned error can't find the container with id edda3768eeeaa0dcd809771dbf0cda2a67d4a9adae1f988ee4df601164dc39db
Oct 30 21:16:30.549717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.549619    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"]
Oct 30 21:16:30.550839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.550809    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4b9d8f7e-082b-438a-b44f-5020dfdd616a" containerName="revoke-keys"
Oct 30 21:16:30.550839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.550839    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9d8f7e-082b-438a-b44f-5020dfdd616a" containerName="revoke-keys"
Oct 30 21:16:30.551054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.551039    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9d8f7e-082b-438a-b44f-5020dfdd616a" containerName="revoke-keys"
Oct 30 21:16:30.557126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.557066    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.602551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.602511    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"]
Oct 30 21:16:30.662190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.660312    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.662190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.660360    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.662190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.660411    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.662190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.660459    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.662190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.660482    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32638ae5-1e74-47de-b9ed-2c04e3811409-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.662190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.660509    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.662190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.660554    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.662190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.660603    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbhp9\" (UniqueName: \"kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-kube-api-access-pbhp9\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.662190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.660731    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.761974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.761940    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.762118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.762063    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.762118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.762081    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.762118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.762100    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.762237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.762133    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.762237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.762148    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32638ae5-1e74-47de-b9ed-2c04e3811409-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.762237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.762168    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.762237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.762213    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.762405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.762260    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbhp9\" (UniqueName: \"kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-kube-api-access-pbhp9\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.762850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.762824    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32638ae5-1e74-47de-b9ed-2c04e3811409-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.763045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.763029    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.763256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.763235    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.763313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.763298    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.763492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.763384    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.764565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.764547    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.764923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.764902    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.772552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.772524    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.772691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.772651    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbhp9\" (UniqueName: \"kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-kube-api-access-pbhp9\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-t62fx\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:30.944919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.944822    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj" event={"ID":"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f","Type":"ContainerStarted","Data":"6d30f1b545d164381783e94add2e61eb48f27d6d745e1168bc7f521cec6cc466"}
Oct 30 21:16:30.944919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.944882    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj" event={"ID":"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f","Type":"ContainerStarted","Data":"edda3768eeeaa0dcd809771dbf0cda2a67d4a9adae1f988ee4df601164dc39db"}
Oct 30 21:16:30.966760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.966702    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj" podStartSLOduration=2.966688605 podStartE2EDuration="2.966688605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:30.964249666 +0000 UTC m=+904.010808829" watchObservedRunningTime="2026-10-30 21:16:30.966688605 +0000 UTC m=+904.013247749"
Oct 30 21:16:30.969285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:30.969261    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:31.239848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.239794    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"]
Oct 30 21:16:31.248850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.248823    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:31.253440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.253409    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"]
Oct 30 21:16:31.262909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.262885    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"]
Oct 30 21:16:31.263402 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:16:31.263363    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32638ae5_1e74_47de_b9ed_2c04e3811409.slice/crio-225e4adc2ac7f8194a1c17ebb7279c8cad8681232e4fd53b6550456ee9c872e2 WatchSource:0}: Error finding container 225e4adc2ac7f8194a1c17ebb7279c8cad8681232e4fd53b6550456ee9c872e2: Status 404 returned error can't find the container with id 225e4adc2ac7f8194a1c17ebb7279c8cad8681232e4fd53b6550456ee9c872e2
Oct 30 21:16:31.267712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.267653    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:31.267940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.267913    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:31.268126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.268104    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:31.370091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.370058    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e37648a2-6jwhl\" (UID: \"7c8b946a-fa86-416f-8c00-303fff251ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:31.370457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.370296    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e37648a2-6jwhl\" (UID: \"7c8b946a-fa86-416f-8c00-303fff251ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:31.370457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.370414    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrxww\" (UniqueName: \"kubernetes.io/projected/7c8b946a-fa86-416f-8c00-303fff251ca3-kube-api-access-xrxww\") pod \"maas-api-revoke-sub-e37648a2-6jwhl\" (UID: \"7c8b946a-fa86-416f-8c00-303fff251ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:31.473948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.473906    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrxww\" (UniqueName: \"kubernetes.io/projected/7c8b946a-fa86-416f-8c00-303fff251ca3-kube-api-access-xrxww\") pod \"maas-api-revoke-sub-e37648a2-6jwhl\" (UID: \"7c8b946a-fa86-416f-8c00-303fff251ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:31.474675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.474646    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e37648a2-6jwhl\" (UID: \"7c8b946a-fa86-416f-8c00-303fff251ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:31.475805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.475702    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e37648a2-6jwhl\" (UID: \"7c8b946a-fa86-416f-8c00-303fff251ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:31.476146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.476112    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e37648a2-6jwhl\" (UID: \"7c8b946a-fa86-416f-8c00-303fff251ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:31.479651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.479628    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e37648a2-6jwhl\" (UID: \"7c8b946a-fa86-416f-8c00-303fff251ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:31.487479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.485467    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrxww\" (UniqueName: \"kubernetes.io/projected/7c8b946a-fa86-416f-8c00-303fff251ca3-kube-api-access-xrxww\") pod \"maas-api-revoke-sub-e37648a2-6jwhl\" (UID: \"7c8b946a-fa86-416f-8c00-303fff251ca3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:31.571290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.571214    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:31.864047 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:16:31.864019    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8b946a_fa86_416f_8c00_303fff251ca3.slice/crio-6ed9819f91104e285e86dbfff608fc923c8341947ba41f110b11227a78a1a5e6 WatchSource:0}: Error finding container 6ed9819f91104e285e86dbfff608fc923c8341947ba41f110b11227a78a1a5e6: Status 404 returned error can't find the container with id 6ed9819f91104e285e86dbfff608fc923c8341947ba41f110b11227a78a1a5e6
Oct 30 21:16:31.864159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.864051    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"]
Oct 30 21:16:31.957829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.957789    4918 generic.go:356] "Generic (PLEG): container finished" podID="7bc720ed-be58-4f75-a5d2-5599b7ab7f4f" containerID="6d30f1b545d164381783e94add2e61eb48f27d6d745e1168bc7f521cec6cc466" exitCode=0
Oct 30 21:16:31.958059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.958032    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj" event={"ID":"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f","Type":"ContainerDied","Data":"6d30f1b545d164381783e94add2e61eb48f27d6d745e1168bc7f521cec6cc466"}
Oct 30 21:16:31.959735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.959703    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl" event={"ID":"7c8b946a-fa86-416f-8c00-303fff251ca3","Type":"ContainerStarted","Data":"6ed9819f91104e285e86dbfff608fc923c8341947ba41f110b11227a78a1a5e6"}
Oct 30 21:16:31.961972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.961943    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx" event={"ID":"32638ae5-1e74-47de-b9ed-2c04e3811409","Type":"ContainerStarted","Data":"5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f"}
Oct 30 21:16:31.962101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.961995    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx" event={"ID":"32638ae5-1e74-47de-b9ed-2c04e3811409","Type":"ContainerStarted","Data":"225e4adc2ac7f8194a1c17ebb7279c8cad8681232e4fd53b6550456ee9c872e2"}
Oct 30 21:16:31.973026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:31.972980    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:32.021651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.021555    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx" podStartSLOduration=2.021540905 podStartE2EDuration="2.021540905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:32.016211745 +0000 UTC m=+905.062770926" watchObservedRunningTime="2026-10-30 21:16:32.021540905 +0000 UTC m=+905.068100050"
Oct 30 21:16:32.329953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.329922    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"]
Oct 30 21:16:32.334016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.333999    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:32.343346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.343319    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"]
Oct 30 21:16:32.416478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.415321    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5779dc1e-vlncb\" (UID: \"69dd6c25-8430-4b8e-97a7-dba7627858b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:32.416478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.415383    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5779dc1e-vlncb\" (UID: \"69dd6c25-8430-4b8e-97a7-dba7627858b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:32.416478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.415547    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrr2w\" (UniqueName: \"kubernetes.io/projected/69dd6c25-8430-4b8e-97a7-dba7627858b5-kube-api-access-wrr2w\") pod \"maas-api-revoke-sub-5779dc1e-vlncb\" (UID: \"69dd6c25-8430-4b8e-97a7-dba7627858b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:32.528416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.524271    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5779dc1e-vlncb\" (UID: \"69dd6c25-8430-4b8e-97a7-dba7627858b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:32.528416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.524331    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5779dc1e-vlncb\" (UID: \"69dd6c25-8430-4b8e-97a7-dba7627858b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:32.528416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.524431    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrr2w\" (UniqueName: \"kubernetes.io/projected/69dd6c25-8430-4b8e-97a7-dba7627858b5-kube-api-access-wrr2w\") pod \"maas-api-revoke-sub-5779dc1e-vlncb\" (UID: \"69dd6c25-8430-4b8e-97a7-dba7627858b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:32.528416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.528335    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5779dc1e-vlncb\" (UID: \"69dd6c25-8430-4b8e-97a7-dba7627858b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:32.547458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.540893    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5779dc1e-vlncb\" (UID: \"69dd6c25-8430-4b8e-97a7-dba7627858b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:32.547458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.547317    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrr2w\" (UniqueName: \"kubernetes.io/projected/69dd6c25-8430-4b8e-97a7-dba7627858b5-kube-api-access-wrr2w\") pod \"maas-api-revoke-sub-5779dc1e-vlncb\" (UID: \"69dd6c25-8430-4b8e-97a7-dba7627858b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:32.650746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.650712    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:32.953010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.952969    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"]
Oct 30 21:16:32.971043 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.971016    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:16:32.971043 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.971025    4918 generic.go:356] "Generic (PLEG): container finished" podID="7c8b946a-fa86-416f-8c00-303fff251ca3" containerID="178dde54e9c8c03b7b693fe4924844720c22e0e66ca8c4e3f3dfabd5ce2f4571" exitCode=0
Oct 30 21:16:32.971204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.971059    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx" podUID="32638ae5-1e74-47de-b9ed-2c04e3811409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:16:32.971204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:32.971062    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl" event={"ID":"7c8b946a-fa86-416f-8c00-303fff251ca3","Type":"ContainerDied","Data":"178dde54e9c8c03b7b693fe4924844720c22e0e66ca8c4e3f3dfabd5ce2f4571"}
Oct 30 21:16:32.981425 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:16:32.981354    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69dd6c25_8430_4b8e_97a7_dba7627858b5.slice/crio-16f7eb63194e94f03c81574b84c33cfa4d98d4cae8528ab9265ea0f55b65d512 WatchSource:0}: Error finding container 16f7eb63194e94f03c81574b84c33cfa4d98d4cae8528ab9265ea0f55b65d512: Status 404 returned error can't find the container with id 16f7eb63194e94f03c81574b84c33cfa4d98d4cae8528ab9265ea0f55b65d512
Oct 30 21:16:33.160482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.160458    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:33.352428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.334909    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-openshift-ca\") pod \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\" (UID: \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\") "
Oct 30 21:16:33.352428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.335142    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-xks-ca\") pod \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\" (UID: \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\") "
Oct 30 21:16:33.352428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.335266    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-kube-api-access-ncr8w\" (UniqueName: \"kubernetes.io/projected/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-kube-api-access-ncr8w\") pod \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\" (UID: \"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f\") "
Oct 30 21:16:33.364456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.361430    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-openshift-ca" pod "7bc720ed-be58-4f75-a5d2-5599b7ab7f4f" (UID: "7bc720ed-be58-4f75-a5d2-5599b7ab7f4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:33.368920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.368886    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-kube-api-access-ncr8w" pod "7bc720ed-be58-4f75-a5d2-5599b7ab7f4f" (UID: "7bc720ed-be58-4f75-a5d2-5599b7ab7f4f"). InnerVolumeSpecName "kube-api-access-ncr8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:33.369733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.369697    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-xks-ca" pod "7bc720ed-be58-4f75-a5d2-5599b7ab7f4f" (UID: "7bc720ed-be58-4f75-a5d2-5599b7ab7f4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:33.436873 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.436846    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:33.436873 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.436873    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncr8w\" (UniqueName: \"kubernetes.io/projected/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-kube-api-access-ncr8w\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:33.437238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.436885    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:33.970072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.970039    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:16:33.970219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.970092    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx" podUID="32638ae5-1e74-47de-b9ed-2c04e3811409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:16:33.981349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.981319    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb" event={"ID":"69dd6c25-8430-4b8e-97a7-dba7627858b5","Type":"ContainerStarted","Data":"33debf2c280668a06e042c34ff180048310b854aa53bdee4628bba2ba391aa74"}
Oct 30 21:16:33.981544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.981371    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb" event={"ID":"69dd6c25-8430-4b8e-97a7-dba7627858b5","Type":"ContainerStarted","Data":"16f7eb63194e94f03c81574b84c33cfa4d98d4cae8528ab9265ea0f55b65d512"}
Oct 30 21:16:33.982870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.982851    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"
Oct 30 21:16:33.982990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.982902    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj" event={"ID":"7bc720ed-be58-4f75-a5d2-5599b7ab7f4f","Type":"ContainerDied","Data":"edda3768eeeaa0dcd809771dbf0cda2a67d4a9adae1f988ee4df601164dc39db"}
Oct 30 21:16:33.982990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:33.982946    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edda3768eeeaa0dcd809771dbf0cda2a67d4a9adae1f988ee4df601164dc39db"
Oct 30 21:16:34.216617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.216551    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb" podStartSLOduration=2.216533069 podStartE2EDuration="2.216533069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:34.008561576 +0000 UTC m=+907.055120754" watchObservedRunningTime="2026-10-30 21:16:34.216533069 +0000 UTC m=+907.263092216"
Oct 30 21:16:34.223969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.223947    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:34.350014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.349986    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-openshift-ca\") pod \"7c8b946a-fa86-416f-8c00-303fff251ca3\" (UID: \"7c8b946a-fa86-416f-8c00-303fff251ca3\") "
Oct 30 21:16:34.350215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.350082    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-xks-ca\") pod \"7c8b946a-fa86-416f-8c00-303fff251ca3\" (UID: \"7c8b946a-fa86-416f-8c00-303fff251ca3\") "
Oct 30 21:16:34.350215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.350177    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c8b946a-fa86-416f-8c00-303fff251ca3-kube-api-access-xrxww\" (UniqueName: \"kubernetes.io/projected/7c8b946a-fa86-416f-8c00-303fff251ca3-kube-api-access-xrxww\") pod \"7c8b946a-fa86-416f-8c00-303fff251ca3\" (UID: \"7c8b946a-fa86-416f-8c00-303fff251ca3\") "
Oct 30 21:16:34.350837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.350807    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-openshift-ca" pod "7c8b946a-fa86-416f-8c00-303fff251ca3" (UID: "7c8b946a-fa86-416f-8c00-303fff251ca3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:34.352790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.352767    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-xks-ca" pod "7c8b946a-fa86-416f-8c00-303fff251ca3" (UID: "7c8b946a-fa86-416f-8c00-303fff251ca3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:34.352857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.352828    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8b946a-fa86-416f-8c00-303fff251ca3-kube-api-access-xrxww" pod "7c8b946a-fa86-416f-8c00-303fff251ca3" (UID: "7c8b946a-fa86-416f-8c00-303fff251ca3"). InnerVolumeSpecName "kube-api-access-xrxww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:34.452108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.452061    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrxww\" (UniqueName: \"kubernetes.io/projected/7c8b946a-fa86-416f-8c00-303fff251ca3-kube-api-access-xrxww\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:34.452108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.452104    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:34.452108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.452115    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c8b946a-fa86-416f-8c00-303fff251ca3-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:34.970114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.970081    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:16:34.970295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:34.970148    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx" podUID="32638ae5-1e74-47de-b9ed-2c04e3811409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:16:35.016771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:35.016743    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"
Oct 30 21:16:35.016976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:35.016741    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl" event={"ID":"7c8b946a-fa86-416f-8c00-303fff251ca3","Type":"ContainerDied","Data":"6ed9819f91104e285e86dbfff608fc923c8341947ba41f110b11227a78a1a5e6"}
Oct 30 21:16:35.016976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:35.016857    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ed9819f91104e285e86dbfff608fc923c8341947ba41f110b11227a78a1a5e6"
Oct 30 21:16:35.018731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:35.018705    4918 generic.go:356] "Generic (PLEG): container finished" podID="69dd6c25-8430-4b8e-97a7-dba7627858b5" containerID="33debf2c280668a06e042c34ff180048310b854aa53bdee4628bba2ba391aa74" exitCode=0
Oct 30 21:16:35.018895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:35.018783    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb" event={"ID":"69dd6c25-8430-4b8e-97a7-dba7627858b5","Type":"ContainerDied","Data":"33debf2c280668a06e042c34ff180048310b854aa53bdee4628bba2ba391aa74"}
Oct 30 21:16:35.969777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:35.969742    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:16:35.970186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:35.969805    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx" podUID="32638ae5-1e74-47de-b9ed-2c04e3811409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:16:36.325744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:36.325718    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:36.379370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:36.379339    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-openshift-ca\") pod \"69dd6c25-8430-4b8e-97a7-dba7627858b5\" (UID: \"69dd6c25-8430-4b8e-97a7-dba7627858b5\") "
Oct 30 21:16:36.379370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:36.379403    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-xks-ca\") pod \"69dd6c25-8430-4b8e-97a7-dba7627858b5\" (UID: \"69dd6c25-8430-4b8e-97a7-dba7627858b5\") "
Oct 30 21:16:36.379668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:36.379576    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69dd6c25-8430-4b8e-97a7-dba7627858b5-kube-api-access-wrr2w\" (UniqueName: \"kubernetes.io/projected/69dd6c25-8430-4b8e-97a7-dba7627858b5-kube-api-access-wrr2w\") pod \"69dd6c25-8430-4b8e-97a7-dba7627858b5\" (UID: \"69dd6c25-8430-4b8e-97a7-dba7627858b5\") "
Oct 30 21:16:36.379869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:36.379832    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-openshift-ca" pod "69dd6c25-8430-4b8e-97a7-dba7627858b5" (UID: "69dd6c25-8430-4b8e-97a7-dba7627858b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:36.380175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:36.380148    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:36.382168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:36.382134    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69dd6c25-8430-4b8e-97a7-dba7627858b5-kube-api-access-wrr2w" pod "69dd6c25-8430-4b8e-97a7-dba7627858b5" (UID: "69dd6c25-8430-4b8e-97a7-dba7627858b5"). InnerVolumeSpecName "kube-api-access-wrr2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:36.384644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:36.384621    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-xks-ca" pod "69dd6c25-8430-4b8e-97a7-dba7627858b5" (UID: "69dd6c25-8430-4b8e-97a7-dba7627858b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:36.481794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:36.481762    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrr2w\" (UniqueName: \"kubernetes.io/projected/69dd6c25-8430-4b8e-97a7-dba7627858b5-kube-api-access-wrr2w\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:36.481794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:36.481791    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69dd6c25-8430-4b8e-97a7-dba7627858b5-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:36.969891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:36.969859    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:16:36.970457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:36.970427    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx" podUID="32638ae5-1e74-47de-b9ed-2c04e3811409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:16:37.075362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:37.075332    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"
Oct 30 21:16:37.079426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:37.079375    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb" event={"ID":"69dd6c25-8430-4b8e-97a7-dba7627858b5","Type":"ContainerDied","Data":"16f7eb63194e94f03c81574b84c33cfa4d98d4cae8528ab9265ea0f55b65d512"}
Oct 30 21:16:37.079587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:37.079449    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16f7eb63194e94f03c81574b84c33cfa4d98d4cae8528ab9265ea0f55b65d512"
Oct 30 21:16:37.970147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:37.970106    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:16:37.970543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:37.970170    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx" podUID="32638ae5-1e74-47de-b9ed-2c04e3811409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:16:38.973969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:38.973930    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:38.974377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:38.974239    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:38.975003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:38.974984    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:16:41.665438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.664655    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"]
Oct 30 21:16:41.668955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.667271    4918 cpu_manager.go:414] "containerMap: removing container" podUID="7c8b946a-fa86-416f-8c00-303fff251ca3" containerName="revoke-keys"
Oct 30 21:16:41.668955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.667297    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c8b946a-fa86-416f-8c00-303fff251ca3" containerName="revoke-keys"
Oct 30 21:16:41.668955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.667407    4918 cpu_manager.go:414] "containerMap: removing container" podUID="7bc720ed-be58-4f75-a5d2-5599b7ab7f4f" containerName="revoke-keys"
Oct 30 21:16:41.668955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.667416    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc720ed-be58-4f75-a5d2-5599b7ab7f4f" containerName="revoke-keys"
Oct 30 21:16:41.668955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.667492    4918 cpu_manager.go:414] "containerMap: removing container" podUID="69dd6c25-8430-4b8e-97a7-dba7627858b5" containerName="revoke-keys"
Oct 30 21:16:41.668955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.667500    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69dd6c25-8430-4b8e-97a7-dba7627858b5" containerName="revoke-keys"
Oct 30 21:16:41.668955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.667763    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c8b946a-fa86-416f-8c00-303fff251ca3" containerName="revoke-keys"
Oct 30 21:16:41.668955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.667785    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc720ed-be58-4f75-a5d2-5599b7ab7f4f" containerName="revoke-keys"
Oct 30 21:16:41.668955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.667795    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="69dd6c25-8430-4b8e-97a7-dba7627858b5" containerName="revoke-keys"
Oct 30 21:16:41.762137 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.762113    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"]
Oct 30 21:16:41.762345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.762248    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:41.765977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.765952    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:16:41.766239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.766223    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:16:41.798476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.798451    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:16:41.798747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.798732    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:16:41.800429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.800410    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4faa5985-5nzkj\" (UID: \"9a6dac54-3b54-403c-991c-4dcc528478f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:41.800514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.800479    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxrxp\" (UniqueName: \"kubernetes.io/projected/9a6dac54-3b54-403c-991c-4dcc528478f6-kube-api-access-bxrxp\") pod \"maas-api-revoke-sub-4faa5985-5nzkj\" (UID: \"9a6dac54-3b54-403c-991c-4dcc528478f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:41.800596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.800583    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4faa5985-5nzkj\" (UID: \"9a6dac54-3b54-403c-991c-4dcc528478f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:41.902148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.902118    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4faa5985-5nzkj\" (UID: \"9a6dac54-3b54-403c-991c-4dcc528478f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:41.902287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.902229    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4faa5985-5nzkj\" (UID: \"9a6dac54-3b54-403c-991c-4dcc528478f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:41.902287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.902278    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxrxp\" (UniqueName: \"kubernetes.io/projected/9a6dac54-3b54-403c-991c-4dcc528478f6-kube-api-access-bxrxp\") pod \"maas-api-revoke-sub-4faa5985-5nzkj\" (UID: \"9a6dac54-3b54-403c-991c-4dcc528478f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:41.903054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.902940    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4faa5985-5nzkj\" (UID: \"9a6dac54-3b54-403c-991c-4dcc528478f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:41.904715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.904691    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4faa5985-5nzkj\" (UID: \"9a6dac54-3b54-403c-991c-4dcc528478f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:41.911022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:41.910996    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxrxp\" (UniqueName: \"kubernetes.io/projected/9a6dac54-3b54-403c-991c-4dcc528478f6-kube-api-access-bxrxp\") pod \"maas-api-revoke-sub-4faa5985-5nzkj\" (UID: \"9a6dac54-3b54-403c-991c-4dcc528478f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:42.077761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:42.077730    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:42.459883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:42.459850    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"]
Oct 30 21:16:43.210408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:43.209591    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj" event={"ID":"9a6dac54-3b54-403c-991c-4dcc528478f6","Type":"ContainerStarted","Data":"17dfb9453ae483209de5219919973daf2de7fdea409c70e8f14fde6d0181b602"}
Oct 30 21:16:43.210408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:43.209653    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj" event={"ID":"9a6dac54-3b54-403c-991c-4dcc528478f6","Type":"ContainerStarted","Data":"73e1409bec24ac953b9558f67f86afef3a35e9b3d29a45165ac8c643d0c6d9bc"}
Oct 30 21:16:43.233939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:43.233876    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj" podStartSLOduration=2.233855153 podStartE2EDuration="2.233855153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:43.226743845 +0000 UTC m=+916.273302991" watchObservedRunningTime="2026-10-30 21:16:43.233855153 +0000 UTC m=+916.280414316"
Oct 30 21:16:44.281416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:44.280503    4918 generic.go:356] "Generic (PLEG): container finished" podID="9a6dac54-3b54-403c-991c-4dcc528478f6" containerID="17dfb9453ae483209de5219919973daf2de7fdea409c70e8f14fde6d0181b602" exitCode=0
Oct 30 21:16:44.281416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:44.280775    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj" event={"ID":"9a6dac54-3b54-403c-991c-4dcc528478f6","Type":"ContainerDied","Data":"17dfb9453ae483209de5219919973daf2de7fdea409c70e8f14fde6d0181b602"}
Oct 30 21:16:44.586483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:44.575347    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:44.643604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:44.643545    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:44.666872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:44.666720    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:44.667883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:44.667831    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:44.727436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:44.716500    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:44.837179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:44.837072    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:16:44.844551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:44.844509    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:16:44.874576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:44.874530    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:44.937465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:44.937415    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:45.101374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.009216    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:45.588608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.588529    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:16:45.613431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.605501    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:16:45.613431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.605889    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Oct 30 21:16:45.613431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.607175    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:16:45.616932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.615609    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:16:45.689420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.669116    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:16:45.689420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.669453    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:45.689420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.669511    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:45.689420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.669809    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:16:45.689420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.669876    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:16:45.723676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.723640    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:16:45.767239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.732074    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"]
Oct 30 21:16:45.808443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.732118    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:45.888471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.858529    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:45.984430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.885479    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:45.984430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.916480    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:45.984430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.938043    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.051419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.937923    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:16:46.051419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.995159    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:16:46.051419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.997105    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:16:46.051419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:45.997949    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:16:46.051419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.011526    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Oct 30 21:16:46.200477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.199674    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:46.205998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.205575    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:16:46.217417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.209688    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:16:46.318733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.318704    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:16:46.405451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.282546    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.405451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.399946    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.405451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.400010    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.405451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.400082    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.405451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.400114    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.540941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.539168    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.540941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.539280    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv68c\" (UniqueName: \"kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-kube-api-access-tv68c\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.540941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.539330    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.540941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.539432    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.595444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.594834    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:46.622930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.622887    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:46.624319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.624286    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:16:46.691375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.624523    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:16:46.691375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.687541    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:16:46.691375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.690874    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.691375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.690931    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv68c\" (UniqueName: \"kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-kube-api-access-tv68c\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.691375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.690974    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.691375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.691024    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.691375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.691081    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.691375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.691123    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.691375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.691182    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.691375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.691243    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.691375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.691272    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.703584 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.693561    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.705597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.703850    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:46.705597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.704974    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:16:46.733936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.733896    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.803054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.786655    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:46.819872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.819842    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.848505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.848214    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv68c\" (UniqueName: \"kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-kube-api-access-tv68c\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.970647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.965554    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.970647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.966368    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:46.970647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:46.966656    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:47.011284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.011235    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.012315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.012284    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:16:47.013358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.012485    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:16:47.048745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.048708    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"]
Oct 30 21:16:47.257435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.208085    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.527611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.235484    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.527611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.504687    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:47.603260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.603209    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.648416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.640632    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:47.659424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.643416    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:16:47.918252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.915803    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:47.928286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.915886    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:47.935416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.915932    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:16:47.936884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.916772    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:47.937052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.937027    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" podUID="6521302f-c44e-40f0-b342-10c81833e27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.937171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.917563    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:47.953494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:47.941696    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.057418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.021030    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:16:48.097415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.023594    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:16:48.162899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.025359    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.514423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.504710    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:48.514423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.504803    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.535419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.519203    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.535419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.520518    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:16:48.535419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.520546    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:16:48.535419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.520614    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.611755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.611703    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.862481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.824189    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:48.924257 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.924214    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.967769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.939125    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.970456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.939741    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:48.970965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.939781    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:48.971537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.956448    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:49.006416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.995550    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-xks-ca\") pod \"9a6dac54-3b54-403c-991c-4dcc528478f6\" (UID: \"9a6dac54-3b54-403c-991c-4dcc528478f6\") "
Oct 30 21:16:49.006416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.995795    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6dac54-3b54-403c-991c-4dcc528478f6-kube-api-access-bxrxp\" (UniqueName: \"kubernetes.io/projected/9a6dac54-3b54-403c-991c-4dcc528478f6-kube-api-access-bxrxp\") pod \"9a6dac54-3b54-403c-991c-4dcc528478f6\" (UID: \"9a6dac54-3b54-403c-991c-4dcc528478f6\") "
Oct 30 21:16:49.006416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.995860    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-openshift-ca\") pod \"9a6dac54-3b54-403c-991c-4dcc528478f6\" (UID: \"9a6dac54-3b54-403c-991c-4dcc528478f6\") "
Oct 30 21:16:49.006416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:48.998637    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-openshift-ca" pod "9a6dac54-3b54-403c-991c-4dcc528478f6" (UID: "9a6dac54-3b54-403c-991c-4dcc528478f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:16:49.358418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.354329    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:16:49.480424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.438244    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:49.604418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.567735    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:16:49.629469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.604913    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": context deadline exceeded"
Oct 30 21:16:49.657415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.633001    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:16:49.657415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.652192    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-xks-ca" pod "9a6dac54-3b54-403c-991c-4dcc528478f6" (UID: "9a6dac54-3b54-403c-991c-4dcc528478f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:16:49.657415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.654604    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:16:49.657415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.583526    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:49.707418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.682208    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:49.788472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.717304    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a6dac54-3b54-403c-991c-4dcc528478f6-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:49.792454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.740613    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6dac54-3b54-403c-991c-4dcc528478f6-kube-api-access-bxrxp" pod "9a6dac54-3b54-403c-991c-4dcc528478f6" (UID: "9a6dac54-3b54-403c-991c-4dcc528478f6"). InnerVolumeSpecName "kube-api-access-bxrxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:16:49.972481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.896978    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxrxp\" (UniqueName: \"kubernetes.io/projected/9a6dac54-3b54-403c-991c-4dcc528478f6-kube-api-access-bxrxp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:16:50.539993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.975527    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:50.612471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.975562    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:16:51.017058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.975597    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:51.210249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.975614    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:51.210249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:49.975649    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:16:51.232076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:50.112969    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"
Oct 30 21:16:51.232596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:50.130187    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj" event={"ID":"9a6dac54-3b54-403c-991c-4dcc528478f6","Type":"ContainerDied","Data":"73e1409bec24ac953b9558f67f86afef3a35e9b3d29a45165ac8c643d0c6d9bc"}
Oct 30 21:16:51.232800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.232781    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73e1409bec24ac953b9558f67f86afef3a35e9b3d29a45165ac8c643d0c6d9bc"
Oct 30 21:16:51.278488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:50.539918    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:51.278488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:50.716107    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:51.278488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:50.716165    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:16:51.278488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:50.359734    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:51.278488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:50.716371    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:16:51.282177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:50.718231    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded"
Oct 30 21:16:51.282426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:50.750239    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 30 21:16:51.282558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:50.870255    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:51.282776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:50.728895    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:16:51.282995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.133330    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:51.659465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.604376    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:16:51.659465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.613943    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:51.659465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.653325    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:51.659465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.653452    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:51.659465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.653498    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:51.693661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.687299    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:16:51.693661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.687497    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"]
Oct 30 21:16:51.693661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.692859    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9a6dac54-3b54-403c-991c-4dcc528478f6" containerName="revoke-keys"
Oct 30 21:16:51.693661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.692882    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6dac54-3b54-403c-991c-4dcc528478f6" containerName="revoke-keys"
Oct 30 21:16:51.741429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.718682    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6dac54-3b54-403c-991c-4dcc528478f6" containerName="revoke-keys"
Oct 30 21:16:51.902621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.902582    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"]
Oct 30 21:16:51.935082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.935010    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"
Oct 30 21:16:52.018492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.947739    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5bc8c60a\""
Oct 30 21:16:52.018492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.960911    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"]
Oct 30 21:16:52.018492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.966795    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5bc8c60a-dockercfg-8k6hk\""
Oct 30 21:16:52.018492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.978455    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:52.018492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:51.982934    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"
Oct 30 21:16:52.018492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.005371    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad901042-4981-479b-98a3-8c6b08db50a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf\" (UID: \"ad901042-4981-479b-98a3-8c6b08db50a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"
Oct 30 21:16:52.018492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.005516    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqm82\" (UniqueName: \"kubernetes.io/projected/ad901042-4981-479b-98a3-8c6b08db50a2-kube-api-access-bqm82\") pod \"payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf\" (UID: \"ad901042-4981-479b-98a3-8c6b08db50a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"
Oct 30 21:16:52.129880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.114686    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad901042-4981-479b-98a3-8c6b08db50a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf\" (UID: \"ad901042-4981-479b-98a3-8c6b08db50a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"
Oct 30 21:16:52.129880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.114834    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqm82\" (UniqueName: \"kubernetes.io/projected/ad901042-4981-479b-98a3-8c6b08db50a2-kube-api-access-bqm82\") pod \"payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf\" (UID: \"ad901042-4981-479b-98a3-8c6b08db50a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"
Oct 30 21:16:52.129880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.114909    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh8lf\" (UniqueName: \"kubernetes.io/projected/1495e8dd-272b-4be0-882e-5d89f2b5223a-kube-api-access-mh8lf\") pod \"payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb\" (UID: \"1495e8dd-272b-4be0-882e-5d89f2b5223a\") " pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"
Oct 30 21:16:52.129880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.114986    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1495e8dd-272b-4be0-882e-5d89f2b5223a-plugins-config-volume\") pod \"payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb\" (UID: \"1495e8dd-272b-4be0-882e-5d89f2b5223a\") " pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"
Oct 30 21:16:52.169083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.134406    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad901042-4981-479b-98a3-8c6b08db50a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf\" (UID: \"ad901042-4981-479b-98a3-8c6b08db50a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"
Oct 30 21:16:52.175726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.175685    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:16:52.208510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.205024    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:52.208510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.205426    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:16:52.225423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.217630    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8lf\" (UniqueName: \"kubernetes.io/projected/1495e8dd-272b-4be0-882e-5d89f2b5223a-kube-api-access-mh8lf\") pod \"payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb\" (UID: \"1495e8dd-272b-4be0-882e-5d89f2b5223a\") " pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"
Oct 30 21:16:52.325539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.325504    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1495e8dd-272b-4be0-882e-5d89f2b5223a-plugins-config-volume\") pod \"payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb\" (UID: \"1495e8dd-272b-4be0-882e-5d89f2b5223a\") " pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"
Oct 30 21:16:52.325938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.272828    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:52.326364 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.272784    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:52.327622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.327598    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1495e8dd-272b-4be0-882e-5d89f2b5223a-plugins-config-volume\") pod \"payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb\" (UID: \"1495e8dd-272b-4be0-882e-5d89f2b5223a\") " pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"
Oct 30 21:16:52.327772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.283507    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:52.328103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.272875    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"]
Oct 30 21:16:52.328264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.308505    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:52.349884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.275238    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh8lf\" (UniqueName: \"kubernetes.io/projected/1495e8dd-272b-4be0-882e-5d89f2b5223a-kube-api-access-mh8lf\") pod \"payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb\" (UID: \"1495e8dd-272b-4be0-882e-5d89f2b5223a\") " pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"
Oct 30 21:16:52.350080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.283548    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:52.370295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.370256    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"
Oct 30 21:16:52.389509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.389474    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"]
Oct 30 21:16:52.463913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.462191    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqm82\" (UniqueName: \"kubernetes.io/projected/ad901042-4981-479b-98a3-8c6b08db50a2-kube-api-access-bqm82\") pod \"payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf\" (UID: \"ad901042-4981-479b-98a3-8c6b08db50a2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"
Oct 30 21:16:52.505605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.505576    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:16:52.619843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.619804    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:16:52.648028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.647996    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:16:52.712424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.711472    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:16:52.785276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.784979    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:16:52.845453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.838636    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"
Oct 30 21:16:52.860418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.857602    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:16:52.880778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.880748    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:16:52.889610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.885197    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:16:52.894444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.892681    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:16:52.894444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.892757    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:16:52.894444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.892812    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:16:52.956413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.950122    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" event={"ID":"c40df71b-5ccf-4655-bee6-eec8cf48d47b","Type":"ContainerStarted","Data":"0c009014a650a262dd529cbf450faa3af5dbd4e826f04a29bcfcb6d5124f8635"}
Oct 30 21:16:52.956413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:52.952239    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:16:53.070081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:53.067983    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"]
Oct 30 21:16:53.404550 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:16:53.404516    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad901042_4981_479b_98a3_8c6b08db50a2.slice/crio-f8234f853dd936e5c1b856d1117c96e489bfbd47ca6964ace0dcfeb1923dc478 WatchSource:0}: Error finding container f8234f853dd936e5c1b856d1117c96e489bfbd47ca6964ace0dcfeb1923dc478: Status 404 returned error can't find the container with id f8234f853dd936e5c1b856d1117c96e489bfbd47ca6964ace0dcfeb1923dc478
Oct 30 21:16:53.452280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:53.452247    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"]
Oct 30 21:16:53.960623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:53.960588    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb" event={"ID":"1495e8dd-272b-4be0-882e-5d89f2b5223a","Type":"ContainerStarted","Data":"11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5"}
Oct 30 21:16:53.960781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:53.960648    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb" event={"ID":"1495e8dd-272b-4be0-882e-5d89f2b5223a","Type":"ContainerStarted","Data":"25170fd45914144d2464bfd3ff52ab67aaee88148d61784f099bef865d21a027"}
Oct 30 21:16:53.960781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:53.960680    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"
Oct 30 21:16:53.962452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:53.962424    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" event={"ID":"c40df71b-5ccf-4655-bee6-eec8cf48d47b","Type":"ContainerStarted","Data":"424bdf4cd0bd06d47e64d4edc0dbe55f1a175fb3569d820f4478dd9bf5c78585"}
Oct 30 21:16:53.964131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:53.964105    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf" event={"ID":"ad901042-4981-479b-98a3-8c6b08db50a2","Type":"ContainerStarted","Data":"20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca"}
Oct 30 21:16:53.964278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:53.964144    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf" event={"ID":"ad901042-4981-479b-98a3-8c6b08db50a2","Type":"ContainerStarted","Data":"f8234f853dd936e5c1b856d1117c96e489bfbd47ca6964ace0dcfeb1923dc478"}
Oct 30 21:16:53.964278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:53.964183    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"
Oct 30 21:16:54.011629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.011562    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb" podStartSLOduration=4.01154268 podStartE2EDuration="4.01154268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:54.007820203 +0000 UTC m=+927.054379376" watchObservedRunningTime="2026-10-30 21:16:54.01154268 +0000 UTC m=+927.058101826"
Oct 30 21:16:54.044004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.043938    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf" podStartSLOduration=5.043923856 podStartE2EDuration="5.043923856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:54.041307013 +0000 UTC m=+927.087866157" watchObservedRunningTime="2026-10-30 21:16:54.043923856 +0000 UTC m=+927.090483001"
Oct 30 21:16:54.119237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.119177    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" podStartSLOduration=9.119161722 podStartE2EDuration="9.119161722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:54.105890487 +0000 UTC m=+927.152449655" watchObservedRunningTime="2026-10-30 21:16:54.119161722 +0000 UTC m=+927.165720916"
Oct 30 21:16:54.424159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.420563    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"]
Oct 30 21:16:54.425683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.425661    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.443947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.440735    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-86671d-openshift-default-dockercfg-ntwdg\""
Oct 30 21:16:54.443947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.443656    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"]
Oct 30 21:16:54.528507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.528478    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735fc195-18ba-475c-b71a-65711bdc589a-istio-podinfo\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.528507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.528510    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-istio-token\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.528712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.528619    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8tds\" (UniqueName: \"kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-kube-api-access-w8tds\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.528712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.528664    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-data\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.528785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.528771    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-certs\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.528820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.528794    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-credential-socket\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.528867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.528852    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-socket\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.528922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.528910    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-envoy\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.528993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.528982    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735fc195-18ba-475c-b71a-65711bdc589a-istiod-ca-cert\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.637670    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8tds\" (UniqueName: \"kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-kube-api-access-w8tds\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.637774    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-data\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.638000    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-certs\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.638027    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-credential-socket\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.638102    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-socket\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.638237    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-envoy\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.638337    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735fc195-18ba-475c-b71a-65711bdc589a-istiod-ca-cert\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.638468    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735fc195-18ba-475c-b71a-65711bdc589a-istio-podinfo\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.638498    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-istio-token\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.639817    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-data\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.640148    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-socket\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.640457    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-certs\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.640803    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-credential-socket\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.647414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.642473    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735fc195-18ba-475c-b71a-65711bdc589a-istiod-ca-cert\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.692078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.687772    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-istio-token\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.695410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.693793    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-envoy\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.724414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.720351    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735fc195-18ba-475c-b71a-65711bdc589a-istio-podinfo\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.724414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.724342    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8tds\" (UniqueName: \"kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-kube-api-access-w8tds\") pod \"e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") " pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.756415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.755666    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:54.928435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.920753    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:16:54.928435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.923608    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:16:54.928435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:54.923662    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" podUID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 21:16:55.221508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:55.221294    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:55.221508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:55.221384    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:55.222568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:55.221794    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:16:55.225321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:55.224342    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"]
Oct 30 21:16:55.917232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:55.917200    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:16:55.917478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:55.917251    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" podUID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 21:16:56.010419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:56.009860    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" event={"ID":"735fc195-18ba-475c-b71a-65711bdc589a","Type":"ContainerStarted","Data":"35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f"}
Oct 30 21:16:56.010419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:56.009960    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" event={"ID":"735fc195-18ba-475c-b71a-65711bdc589a","Type":"ContainerStarted","Data":"30b79ca700a2f990e935ab94b43354417e7c5b612d7933486322c4e85bce45f2"}
Oct 30 21:16:56.032215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:56.032168    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" podStartSLOduration=2.032154966 podStartE2EDuration="2.032154966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:56.030828975 +0000 UTC m=+929.077388145" watchObservedRunningTime="2026-10-30 21:16:56.032154966 +0000 UTC m=+929.078714111"
Oct 30 21:16:56.757498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:56.757468    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:56.763087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:56.763061    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:56.918084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:56.918051    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:16:56.918252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:56.918104    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" podUID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 21:16:57.027806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:57.027710    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:57.029007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:57.028982    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:16:57.918026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:57.917994    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:16:57.918539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:57.918050    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" podUID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 21:16:58.917202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:58.917169    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:16:58.917477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:58.917453    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" podUID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 21:16:59.161745 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:16:59.161709    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:16:59.745611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:59.745580    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"]
Oct 30 21:16:59.750577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:59.750559    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"
Oct 30 21:16:59.759867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:59.759838    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"]
Oct 30 21:16:59.761799 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:59.761777    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-86671d\""
Oct 30 21:16:59.917418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:59.898496    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea25e13-be53-4cce-81bf-2ff2aa342f3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-86671d-57c7476599-fwsr8\" (UID: \"cea25e13-be53-4cce-81bf-2ff2aa342f3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"
Oct 30 21:16:59.917418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:16:59.898792    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dls5\" (UniqueName: \"kubernetes.io/projected/cea25e13-be53-4cce-81bf-2ff2aa342f3b-kube-api-access-6dls5\") pod \"payload-processing-e2e-api-a-86671d-57c7476599-fwsr8\" (UID: \"cea25e13-be53-4cce-81bf-2ff2aa342f3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"
Oct 30 21:17:00.061472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:00.042465    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea25e13-be53-4cce-81bf-2ff2aa342f3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-86671d-57c7476599-fwsr8\" (UID: \"cea25e13-be53-4cce-81bf-2ff2aa342f3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"
Oct 30 21:17:00.107016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:00.068685    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Oct 30 21:17:00.113841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:00.113785    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" podUID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Oct 30 21:17:00.114078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:00.111802    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dls5\" (UniqueName: \"kubernetes.io/projected/cea25e13-be53-4cce-81bf-2ff2aa342f3b-kube-api-access-6dls5\") pod \"payload-processing-e2e-api-a-86671d-57c7476599-fwsr8\" (UID: \"cea25e13-be53-4cce-81bf-2ff2aa342f3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"
Oct 30 21:17:00.114443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:00.082729    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea25e13-be53-4cce-81bf-2ff2aa342f3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-86671d-57c7476599-fwsr8\" (UID: \"cea25e13-be53-4cce-81bf-2ff2aa342f3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"
Oct 30 21:17:00.163165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:00.163132    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dls5\" (UniqueName: \"kubernetes.io/projected/cea25e13-be53-4cce-81bf-2ff2aa342f3b-kube-api-access-6dls5\") pod \"payload-processing-e2e-api-a-86671d-57c7476599-fwsr8\" (UID: \"cea25e13-be53-4cce-81bf-2ff2aa342f3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"
Oct 30 21:17:00.370848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:00.370773    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"
Oct 30 21:17:00.781667 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:17:00.781629    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea25e13_be53_4cce_81bf_2ff2aa342f3b.slice/crio-be51be62b5bc65edae66ccd3c7b8b0d111d9cd2e93ff2e79a5d866d5db6d3659 WatchSource:0}: Error finding container be51be62b5bc65edae66ccd3c7b8b0d111d9cd2e93ff2e79a5d866d5db6d3659: Status 404 returned error can't find the container with id be51be62b5bc65edae66ccd3c7b8b0d111d9cd2e93ff2e79a5d866d5db6d3659
Oct 30 21:17:00.785846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:00.785820    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"]
Oct 30 21:17:01.202062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:01.201955    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8" event={"ID":"cea25e13-be53-4cce-81bf-2ff2aa342f3b","Type":"ContainerStarted","Data":"074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d"}
Oct 30 21:17:01.202062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:01.202051    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8" event={"ID":"cea25e13-be53-4cce-81bf-2ff2aa342f3b","Type":"ContainerStarted","Data":"be51be62b5bc65edae66ccd3c7b8b0d111d9cd2e93ff2e79a5d866d5db6d3659"}
Oct 30 21:17:01.202648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:01.202084    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"
Oct 30 21:17:01.222374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:01.222325    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8" podStartSLOduration=2.222312942 podStartE2EDuration="2.222312942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:01.218992457 +0000 UTC m=+934.265551629" watchObservedRunningTime="2026-10-30 21:17:01.222312942 +0000 UTC m=+934.268872087"
Oct 30 21:17:01.293648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:01.293601    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:17:01.294046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:01.294021    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:17:01.295335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:01.295303    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:17:03.658643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:03.658593    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:03.865691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:03.864503    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:03.936032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:03.935943    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:17:04.022419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:04.019570    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:04.022419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:04.020755    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:04.022419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:04.020935    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:17:04.022419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:04.021586    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:17:04.600506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:04.599505    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:04.636586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:04.636532    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:04.663402 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:04.663344    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:04.664506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:04.664485    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:17:04.715514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:04.715467    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:04.966476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:04.957042    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:04.966476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:04.957511    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:17:04.966476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:04.958146    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:17:05.022099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.022060    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:05.030415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.022800    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:17:05.030995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.030977    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"
Oct 30 21:17:05.089426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.087668    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"
Oct 30 21:17:05.521527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.521484    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:05.563942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.563896    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:17:05.654417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.630816    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:05.654417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.647048    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:17:05.676571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.676523    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:05.700617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.693159    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:17:05.717617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.716747    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:17:05.723413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.722827    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:17:05.731769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.730020    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded"
Oct 30 21:17:05.869714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.869604    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:05.957897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.957855    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:05.966433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.958155    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:17:05.966960 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.966937    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:17:05.969512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:05.969478    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:06.086270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.078724    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:17:06.086664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.086627    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:06.088023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.087984    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:06.120172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.111151    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:06.358462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.355251    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:06.358462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.355313    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" podUID="02880084-8170-487e-9fdf-0e9a3c7ba287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:06.581748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.581701    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:06.582578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.582555    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:17:06.707417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.684614    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:17:06.802417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.738505    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:06.802417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.740644    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:17:06.921591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.896375    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:17:07.007485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:06.991450    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:07.029417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.016734    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:17:07.406929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.141650    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:17:07.407600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.141874    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:17:07.543414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.183895    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:17:07.543414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.218071    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:07.543414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.408277    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:17:07.543414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.530458    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:07.543779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.543726    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:07.543884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.543138    4918 patch_prober.go:29] interesting pod/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:07.544044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.544020    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:07.602419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.582497    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:07.605850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.605801    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:07.606024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.605864    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" podUID="197109cc-f204-4d13-bf73-02172b3003ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:07.679468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.678983    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:17:07.679468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.679077    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:17:07.679468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.679137    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:17:07.776414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.775549    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:07.840418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.777836    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:17:07.840418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:07.815557    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:17:08.071435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.071359    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:17:08.106315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.106286    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:17:08.133712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.133670    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:08.424687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.424588    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:08.622837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.622728    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:17:08.653417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.649974    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:08.657512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.657472    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:17:08.658165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.658126    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:08.759426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.757818    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-d54n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:08.759426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.757881    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-d54n7" podUID="40508368-7d69-41e3-a2ee-0be190a20281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:08.781428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.779472    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:08.781428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.780485    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:17:08.781428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.780638    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-w9kjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:17:08.781428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.780775    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-w9kjp" podUID="de5f2249-cc22-496c-b84a-8d1fcf7c5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:08.839469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:08.830889    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:09.090462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.077333    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:09.137917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.132049    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:09.137917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.136142    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:09.367532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.361592    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:17:09.382417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.371546    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:17:09.382417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.371590    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:17:09.382417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.371621    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:17:09.382417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.371652    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:17:09.382417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.371684    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:17:09.630349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.630252    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:17:09.658159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.658119    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:09.796415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.795726    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:17:09.910469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.845457    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:09.910469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:09.850379    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:17:10.024427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:10.023762    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:17:10.041964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:10.041936    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:17:12.241646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:12.241599    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"
Oct 30 21:17:15.686588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.686554    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"]
Oct 30 21:17:15.691078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.691056    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.706430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.704318    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"]
Oct 30 21:17:15.757956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.757918    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.758103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.757969    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82566da-633b-47a2-8b06-389302d80839-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.758146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.758126    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82566da-633b-47a2-8b06-389302d80839-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.758200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.758170    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.758316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.758294    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.758410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.758372    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.758491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.758471    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.758624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.758603    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2jfw\" (UniqueName: \"kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-kube-api-access-t2jfw\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.758674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.758659    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.860295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.860250    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.860528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.860351    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jfw\" (UniqueName: \"kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-kube-api-access-t2jfw\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.860528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.860440    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.860656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.860594    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.860656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.860622    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82566da-633b-47a2-8b06-389302d80839-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.860769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.860713    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.860769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.860721    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82566da-633b-47a2-8b06-389302d80839-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.860853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.860788    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.860853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.860820    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.860981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.860862    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.860981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.860879    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.861200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.861173    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.861287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.861229    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.861352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.861340    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82566da-633b-47a2-8b06-389302d80839-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.863150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.863130    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.863349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.863332    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82566da-633b-47a2-8b06-389302d80839-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.868710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.868690    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:15.868926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:15.868904    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jfw\" (UniqueName: \"kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-kube-api-access-t2jfw\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:16.008078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:16.008037    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:16.302504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:16.301011    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"]
Oct 30 21:17:16.328362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:16.321319    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:17:16.328362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:16.322088    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:17:16.328362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:16.322382    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:17:17.065415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:17.062184    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" event={"ID":"f82566da-633b-47a2-8b06-389302d80839","Type":"ContainerStarted","Data":"5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073"}
Oct 30 21:17:17.065415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:17.062295    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" event={"ID":"f82566da-633b-47a2-8b06-389302d80839","Type":"ContainerStarted","Data":"66986ada6549f4414d9e111462a1a8f87ac42d09aced17d5856602a50e761ff6"}
Oct 30 21:17:18.009418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:18.008564    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:18.010927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:18.010690    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:17:18.010927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:18.010748    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" podUID="f82566da-633b-47a2-8b06-389302d80839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:17:19.010777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:19.010742    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:17:19.011171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:19.010794    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" podUID="f82566da-633b-47a2-8b06-389302d80839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:17:20.011966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:20.011905    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:17:20.012546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:20.011998    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" podUID="f82566da-633b-47a2-8b06-389302d80839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:17:21.009905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:21.009873    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:17:21.010242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:21.010184    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" podUID="f82566da-633b-47a2-8b06-389302d80839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:17:22.009416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:22.008772    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:17:22.009416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:22.008846    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" podUID="f82566da-633b-47a2-8b06-389302d80839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:17:23.008951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:23.008918    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:17:23.009160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:23.008967    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" podUID="f82566da-633b-47a2-8b06-389302d80839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:17:24.396776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:24.396745    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:24.397321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:24.397279    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:24.398151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:24.398130    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:17:24.429829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:24.429768    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" podStartSLOduration=9.429754424 podStartE2EDuration="9.429754424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:17.096835025 +0000 UTC m=+950.143394171" watchObservedRunningTime="2026-10-30 21:17:24.429754424 +0000 UTC m=+957.476313569"
Oct 30 21:17:27.442180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.442149    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"]
Oct 30 21:17:27.447550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.447532    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"
Oct 30 21:17:27.456727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.456694    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"]
Oct 30 21:17:27.457272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.457248    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c499bf\""
Oct 30 21:17:27.457382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.457323    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c499bf-dockercfg-7xtjp\""
Oct 30 21:17:27.527682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.527610    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"]
Oct 30 21:17:27.600423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.598932    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdwjj\" (UniqueName: \"kubernetes.io/projected/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-kube-api-access-pdwjj\") pod \"payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b\" (UID: \"acc8d8d8-6d8c-4c65-adb2-413fa37ee354\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"
Oct 30 21:17:27.600423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.599119    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b\" (UID: \"acc8d8d8-6d8c-4c65-adb2-413fa37ee354\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"
Oct 30 21:17:27.613650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.604423    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"
Oct 30 21:17:27.706615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.706529    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbff0160-e9cf-4b95-848d-0336168b038c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9\" (UID: \"bbff0160-e9cf-4b95-848d-0336168b038c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"
Oct 30 21:17:27.706615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.706591    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b\" (UID: \"acc8d8d8-6d8c-4c65-adb2-413fa37ee354\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"
Oct 30 21:17:27.706839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.706647    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshfd\" (UniqueName: \"kubernetes.io/projected/bbff0160-e9cf-4b95-848d-0336168b038c-kube-api-access-cshfd\") pod \"payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9\" (UID: \"bbff0160-e9cf-4b95-848d-0336168b038c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"
Oct 30 21:17:27.706895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.706881    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdwjj\" (UniqueName: \"kubernetes.io/projected/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-kube-api-access-pdwjj\") pod \"payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b\" (UID: \"acc8d8d8-6d8c-4c65-adb2-413fa37ee354\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"
Oct 30 21:17:27.748470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.708286    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b\" (UID: \"acc8d8d8-6d8c-4c65-adb2-413fa37ee354\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"
Oct 30 21:17:27.850513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.850419    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"]
Oct 30 21:17:27.861728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.859838    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbff0160-e9cf-4b95-848d-0336168b038c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9\" (UID: \"bbff0160-e9cf-4b95-848d-0336168b038c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"
Oct 30 21:17:27.861728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.859954    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshfd\" (UniqueName: \"kubernetes.io/projected/bbff0160-e9cf-4b95-848d-0336168b038c-kube-api-access-cshfd\") pod \"payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9\" (UID: \"bbff0160-e9cf-4b95-848d-0336168b038c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"
Oct 30 21:17:27.861728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.861238    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbff0160-e9cf-4b95-848d-0336168b038c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9\" (UID: \"bbff0160-e9cf-4b95-848d-0336168b038c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"
Oct 30 21:17:27.871453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.869306    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdwjj\" (UniqueName: \"kubernetes.io/projected/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-kube-api-access-pdwjj\") pod \"payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b\" (UID: \"acc8d8d8-6d8c-4c65-adb2-413fa37ee354\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"
Oct 30 21:17:27.871453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:27.870695    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshfd\" (UniqueName: \"kubernetes.io/projected/bbff0160-e9cf-4b95-848d-0336168b038c-kube-api-access-cshfd\") pod \"payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9\" (UID: \"bbff0160-e9cf-4b95-848d-0336168b038c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"
Oct 30 21:17:28.081956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:28.081922    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"
Oct 30 21:17:28.168858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:28.168817    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"
Oct 30 21:17:28.416341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:28.416253    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"]
Oct 30 21:17:28.420149 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:17:28.420111    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc8d8d8_6d8c_4c65_adb2_413fa37ee354.slice/crio-4e9a59238fad32c702b191d8ad63ec79890d5b6afea6eab3d442756ce4c35bd9 WatchSource:0}: Error finding container 4e9a59238fad32c702b191d8ad63ec79890d5b6afea6eab3d442756ce4c35bd9: Status 404 returned error can't find the container with id 4e9a59238fad32c702b191d8ad63ec79890d5b6afea6eab3d442756ce4c35bd9
Oct 30 21:17:28.420360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:28.420337    4918 scope.go:122] "RemoveContainer" containerID="4fc5f3eaf29f03374805b998e6106f89c53d013130726151851d681dd49b11d8"
Oct 30 21:17:28.547993 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:17:28.547961    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbff0160_e9cf_4b95_848d_0336168b038c.slice/crio-3cf05e9568aeaa076ffda237cf1aae5daf8896799a0809b4414702de6b1a2789 WatchSource:0}: Error finding container 3cf05e9568aeaa076ffda237cf1aae5daf8896799a0809b4414702de6b1a2789: Status 404 returned error can't find the container with id 3cf05e9568aeaa076ffda237cf1aae5daf8896799a0809b4414702de6b1a2789
Oct 30 21:17:28.550173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:28.550145    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"]
Oct 30 21:17:29.277683 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:17:29.277648    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:17:29.375354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:29.375321    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b" event={"ID":"acc8d8d8-6d8c-4c65-adb2-413fa37ee354","Type":"ContainerStarted","Data":"f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f"}
Oct 30 21:17:29.375575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:29.375375    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b" event={"ID":"acc8d8d8-6d8c-4c65-adb2-413fa37ee354","Type":"ContainerStarted","Data":"4e9a59238fad32c702b191d8ad63ec79890d5b6afea6eab3d442756ce4c35bd9"}
Oct 30 21:17:29.375575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:29.375440    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"
Oct 30 21:17:29.377119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:29.377091    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9" event={"ID":"bbff0160-e9cf-4b95-848d-0336168b038c","Type":"ContainerStarted","Data":"d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984"}
Oct 30 21:17:29.377248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:29.377141    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9" event={"ID":"bbff0160-e9cf-4b95-848d-0336168b038c","Type":"ContainerStarted","Data":"3cf05e9568aeaa076ffda237cf1aae5daf8896799a0809b4414702de6b1a2789"}
Oct 30 21:17:29.377248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:29.377202    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"
Oct 30 21:17:29.414564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:29.414502    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b" podStartSLOduration=2.414482624 podStartE2EDuration="2.414482624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:29.410719534 +0000 UTC m=+962.457278679" watchObservedRunningTime="2026-10-30 21:17:29.414482624 +0000 UTC m=+962.461041774"
Oct 30 21:17:29.434745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:29.434692    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9" podStartSLOduration=2.4346795390000002 podStartE2EDuration="2.434679539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:29.432979417 +0000 UTC m=+962.479538585" watchObservedRunningTime="2026-10-30 21:17:29.434679539 +0000 UTC m=+962.481238683"
Oct 30 21:17:35.047718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.047680    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"]
Oct 30 21:17:35.052729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.052705    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.064697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.064678    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"]
Oct 30 21:17:35.065499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.065480    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b3d1de-openshift-default-dockercfg-vctm9\""
Oct 30 21:17:35.152575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.147997    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-credential-socket\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.152575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.148054    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-certs\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.152575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.148126    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istiod-ca-cert\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.152575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.148223    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-podinfo\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.152575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.148281    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-socket\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.152575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.148314    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtkfk\" (UniqueName: \"kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-kube-api-access-rtkfk\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.152575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.148436    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-data\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.152575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.148534    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-token\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.288418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.148607    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-envoy\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.393665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.393570    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-podinfo\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.395668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.395642    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-socket\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.395861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.395844    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtkfk\" (UniqueName: \"kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-kube-api-access-rtkfk\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.396032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.396015    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-data\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.396217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.396197    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-token\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.396404 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.396372    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-envoy\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.411021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.396591    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-credential-socket\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.411274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.411251    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-certs\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.411800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.397549    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-data\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.411916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.398044    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-credential-socket\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.412001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.397077    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-socket\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.412081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.411696    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-certs\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.412249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.412235    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istiod-ca-cert\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.413120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.413019    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istiod-ca-cert\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.425410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.420693    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-podinfo\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.431056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.431028    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtkfk\" (UniqueName: \"kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-kube-api-access-rtkfk\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.434861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.433765    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-token\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.435976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.435953    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-envoy\") pod \"e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") " pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.476166 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.476138    4918 generic.go:356] "Generic (PLEG): container finished" podID="735fc195-18ba-475c-b71a-65711bdc589a" containerID="35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f" exitCode=0
Oct 30 21:17:35.476303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.476205    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" event={"ID":"735fc195-18ba-475c-b71a-65711bdc589a","Type":"ContainerDied","Data":"35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f"}
Oct 30 21:17:35.476660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.476641    4918 scope.go:122] "RemoveContainer" containerID="35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f"
Oct 30 21:17:35.478306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.478276    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:17:35.478411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.478343    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:17:35.478411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.478375    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:17:35.675612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.671684    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:35.989287 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:17:35.989257    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae7a5a7_5cd7_4094_863c_2de252eda90b.slice/crio-3879b70e9f9d4bae5176f73cbfb2a7243e7aeb3864939af251ae1b5693d41aa1 WatchSource:0}: Error finding container 3879b70e9f9d4bae5176f73cbfb2a7243e7aeb3864939af251ae1b5693d41aa1: Status 404 returned error can't find the container with id 3879b70e9f9d4bae5176f73cbfb2a7243e7aeb3864939af251ae1b5693d41aa1
Oct 30 21:17:35.990746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.990722    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"]
Oct 30 21:17:35.992128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.992095    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:17:35.992208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.992161    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:17:35.992208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:35.992190    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:17:36.508414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:36.505417    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" event={"ID":"6ae7a5a7-5cd7-4094-863c-2de252eda90b","Type":"ContainerStarted","Data":"5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec"}
Oct 30 21:17:36.508414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:36.505490    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" event={"ID":"6ae7a5a7-5cd7-4094-863c-2de252eda90b","Type":"ContainerStarted","Data":"3879b70e9f9d4bae5176f73cbfb2a7243e7aeb3864939af251ae1b5693d41aa1"}
Oct 30 21:17:36.511970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:36.511936    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" event={"ID":"735fc195-18ba-475c-b71a-65711bdc589a","Type":"ContainerStarted","Data":"79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5"}
Oct 30 21:17:36.539082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:36.539029    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" podStartSLOduration=1.53900003 podStartE2EDuration="1.53900003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:36.537054232 +0000 UTC m=+969.583613390" watchObservedRunningTime="2026-10-30 21:17:36.53900003 +0000 UTC m=+969.585559174"
Oct 30 21:17:36.681638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:36.681531    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:36.759119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:36.758635    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:17:36.759119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:36.758761    4918 patch_prober.go:29] interesting pod/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 21:17:36.759119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:36.758805    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 21:17:37.681288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:37.681256    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:37.766415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:37.759827    4918 patch_prober.go:29] interesting pod/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 21:17:37.766415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:37.759885    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 21:17:38.532420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:38.532366    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:38.538024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:38.537997    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:17:38.757602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:38.757566    4918 patch_prober.go:29] interesting pod/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 21:17:38.757979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:38.757631    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 21:17:39.757644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:39.757613    4918 patch_prober.go:29] interesting pod/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 21:17:39.758036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:39.757666    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 21:17:40.265450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.265417    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"]
Oct 30 21:17:40.273768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.273742    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"
Oct 30 21:17:40.278699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.278674    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"]
Oct 30 21:17:40.279991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.279968    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b3d1de\""
Oct 30 21:17:40.281095 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.281063    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b3d1de-dockercfg-6jjb6\""
Oct 30 21:17:40.328278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.328243    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"]
Oct 30 21:17:40.334081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.334053    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"
Oct 30 21:17:40.335051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.335026    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"]
Oct 30 21:17:40.381978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.381813    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e099b0-8c43-4d95-b056-fed2463dcd53-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp\" (UID: \"13e099b0-8c43-4d95-b056-fed2463dcd53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"
Oct 30 21:17:40.381978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.381934    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7fr6\" (UniqueName: \"kubernetes.io/projected/13e099b0-8c43-4d95-b056-fed2463dcd53-kube-api-access-x7fr6\") pod \"payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp\" (UID: \"13e099b0-8c43-4d95-b056-fed2463dcd53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"
Oct 30 21:17:40.433186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.432581    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"
Oct 30 21:17:40.475431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.474596    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"
Oct 30 21:17:40.515729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.509722    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h\" (UID: \"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"
Oct 30 21:17:40.515729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.509806    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g6mh\" (UniqueName: \"kubernetes.io/projected/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-kube-api-access-7g6mh\") pod \"payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h\" (UID: \"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"
Oct 30 21:17:40.523414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.519575    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e099b0-8c43-4d95-b056-fed2463dcd53-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp\" (UID: \"13e099b0-8c43-4d95-b056-fed2463dcd53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"
Oct 30 21:17:40.523414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.519773    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7fr6\" (UniqueName: \"kubernetes.io/projected/13e099b0-8c43-4d95-b056-fed2463dcd53-kube-api-access-x7fr6\") pod \"payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp\" (UID: \"13e099b0-8c43-4d95-b056-fed2463dcd53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"
Oct 30 21:17:40.523414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.521217    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e099b0-8c43-4d95-b056-fed2463dcd53-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp\" (UID: \"13e099b0-8c43-4d95-b056-fed2463dcd53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"
Oct 30 21:17:40.582465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.575465    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7fr6\" (UniqueName: \"kubernetes.io/projected/13e099b0-8c43-4d95-b056-fed2463dcd53-kube-api-access-x7fr6\") pod \"payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp\" (UID: \"13e099b0-8c43-4d95-b056-fed2463dcd53\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"
Oct 30 21:17:40.639607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.639563    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"
Oct 30 21:17:40.739021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.734936    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h\" (UID: \"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"
Oct 30 21:17:40.739021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.735039    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g6mh\" (UniqueName: \"kubernetes.io/projected/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-kube-api-access-7g6mh\") pod \"payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h\" (UID: \"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"
Oct 30 21:17:40.739021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.736628    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h\" (UID: \"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"
Oct 30 21:17:40.780512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.774828    4918 patch_prober.go:29] interesting pod/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 21:17:40.780512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.774890    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 21:17:40.838155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.838112    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g6mh\" (UniqueName: \"kubernetes.io/projected/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-kube-api-access-7g6mh\") pod \"payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h\" (UID: \"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4\") " pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"
Oct 30 21:17:40.959118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:40.958777    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"
Oct 30 21:17:41.257106 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:17:41.223901    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e099b0_8c43_4d95_b056_fed2463dcd53.slice/crio-b6957a7893c617a50bf2c82054ef5fe9ba8472927d5f6b1d9a0e221e105c4f78 WatchSource:0}: Error finding container b6957a7893c617a50bf2c82054ef5fe9ba8472927d5f6b1d9a0e221e105c4f78: Status 404 returned error can't find the container with id b6957a7893c617a50bf2c82054ef5fe9ba8472927d5f6b1d9a0e221e105c4f78
Oct 30 21:17:41.326199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:41.326161    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"]
Oct 30 21:17:41.331859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:41.331834    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"]
Oct 30 21:17:41.332404 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:17:41.332321    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef466c2f_9fc8_4ac3_9ded_f89715a5cfa4.slice/crio-9d15f1447fdda49da2b7bc9f4955cd95db30515e71bf9f2fde573af899454cca WatchSource:0}: Error finding container 9d15f1447fdda49da2b7bc9f4955cd95db30515e71bf9f2fde573af899454cca: Status 404 returned error can't find the container with id 9d15f1447fdda49da2b7bc9f4955cd95db30515e71bf9f2fde573af899454cca
Oct 30 21:17:41.758319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:41.758292    4918 patch_prober.go:29] interesting pod/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 30 21:17:41.758501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:41.758339    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 30 21:17:41.850440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:41.850373    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp" event={"ID":"13e099b0-8c43-4d95-b056-fed2463dcd53","Type":"ContainerStarted","Data":"232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8"}
Oct 30 21:17:41.850901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:41.850471    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"
Oct 30 21:17:41.850901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:41.850498    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp" event={"ID":"13e099b0-8c43-4d95-b056-fed2463dcd53","Type":"ContainerStarted","Data":"b6957a7893c617a50bf2c82054ef5fe9ba8472927d5f6b1d9a0e221e105c4f78"}
Oct 30 21:17:41.852140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:41.852113    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h" event={"ID":"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4","Type":"ContainerStarted","Data":"fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a"}
Oct 30 21:17:41.852269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:41.852161    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h" event={"ID":"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4","Type":"ContainerStarted","Data":"9d15f1447fdda49da2b7bc9f4955cd95db30515e71bf9f2fde573af899454cca"}
Oct 30 21:17:41.852269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:41.852186    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"
Oct 30 21:17:41.874271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:41.874217    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp" podStartSLOduration=1.874203741 podStartE2EDuration="1.874203741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:41.871883169 +0000 UTC m=+974.918442336" watchObservedRunningTime="2026-10-30 21:17:41.874203741 +0000 UTC m=+974.920762886"
Oct 30 21:17:41.936353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:41.936288    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h" podStartSLOduration=1.936265946 podStartE2EDuration="1.936265946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:41.903685759 +0000 UTC m=+974.950244906" watchObservedRunningTime="2026-10-30 21:17:41.936265946 +0000 UTC m=+974.982825106"
Oct 30 21:17:42.044468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:42.033571    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:17:42.779264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:42.778853    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:17:42.861495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:42.861470    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:17:48.006123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.006080    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"]
Oct 30 21:17:48.010757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.010736    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:48.017821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.017789    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:17:48.017951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.017857    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:17:48.018445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.018423    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:17:48.018564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.018462    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:17:48.021576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.021554    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"]
Oct 30 21:17:48.165867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.165835    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk\" (UID: \"7abc0d07-ffde-4ca7-80d1-8461012f4706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:48.166026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.165974    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk\" (UID: \"7abc0d07-ffde-4ca7-80d1-8461012f4706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:48.166105 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.166039    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrwfk\" (UniqueName: \"kubernetes.io/projected/7abc0d07-ffde-4ca7-80d1-8461012f4706-kube-api-access-nrwfk\") pod \"maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk\" (UID: \"7abc0d07-ffde-4ca7-80d1-8461012f4706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:48.267433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.267301    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk\" (UID: \"7abc0d07-ffde-4ca7-80d1-8461012f4706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:48.267619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.267460    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk\" (UID: \"7abc0d07-ffde-4ca7-80d1-8461012f4706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:48.267619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.267516    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrwfk\" (UniqueName: \"kubernetes.io/projected/7abc0d07-ffde-4ca7-80d1-8461012f4706-kube-api-access-nrwfk\") pod \"maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk\" (UID: \"7abc0d07-ffde-4ca7-80d1-8461012f4706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:48.268253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.268223    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk\" (UID: \"7abc0d07-ffde-4ca7-80d1-8461012f4706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:48.270191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.270164    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk\" (UID: \"7abc0d07-ffde-4ca7-80d1-8461012f4706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:48.277443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.277414    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrwfk\" (UniqueName: \"kubernetes.io/projected/7abc0d07-ffde-4ca7-80d1-8461012f4706-kube-api-access-nrwfk\") pod \"maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk\" (UID: \"7abc0d07-ffde-4ca7-80d1-8461012f4706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:48.328268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.328245    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:48.660112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:48.659968    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"]
Oct 30 21:17:49.035971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:49.035932    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk" event={"ID":"7abc0d07-ffde-4ca7-80d1-8461012f4706","Type":"ContainerStarted","Data":"c04dbf7af73fc22cf6071080b807c1b2177986463b27c76ce1fdbd99a2561c24"}
Oct 30 21:17:50.045644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:50.045612    4918 generic.go:356] "Generic (PLEG): container finished" podID="7abc0d07-ffde-4ca7-80d1-8461012f4706" containerID="0b4e3c2c11765172d181ec6d64654565fa269bbb575064b711cf3078d6a115b9" exitCode=0
Oct 30 21:17:50.046070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:50.045696    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk" event={"ID":"7abc0d07-ffde-4ca7-80d1-8461012f4706","Type":"ContainerDied","Data":"0b4e3c2c11765172d181ec6d64654565fa269bbb575064b711cf3078d6a115b9"}
Oct 30 21:17:51.318883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:51.318861    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:51.371097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:51.371012    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-xks-ca\") pod \"7abc0d07-ffde-4ca7-80d1-8461012f4706\" (UID: \"7abc0d07-ffde-4ca7-80d1-8461012f4706\") "
Oct 30 21:17:51.371248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:51.371143    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abc0d07-ffde-4ca7-80d1-8461012f4706-kube-api-access-nrwfk\" (UniqueName: \"kubernetes.io/projected/7abc0d07-ffde-4ca7-80d1-8461012f4706-kube-api-access-nrwfk\") pod \"7abc0d07-ffde-4ca7-80d1-8461012f4706\" (UID: \"7abc0d07-ffde-4ca7-80d1-8461012f4706\") "
Oct 30 21:17:51.371415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:51.371379    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-openshift-ca\") pod \"7abc0d07-ffde-4ca7-80d1-8461012f4706\" (UID: \"7abc0d07-ffde-4ca7-80d1-8461012f4706\") "
Oct 30 21:17:51.372023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:51.371994    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-openshift-ca" pod "7abc0d07-ffde-4ca7-80d1-8461012f4706" (UID: "7abc0d07-ffde-4ca7-80d1-8461012f4706"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:17:51.373415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:51.373375    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-xks-ca" pod "7abc0d07-ffde-4ca7-80d1-8461012f4706" (UID: "7abc0d07-ffde-4ca7-80d1-8461012f4706"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:17:51.374004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:51.373984    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abc0d07-ffde-4ca7-80d1-8461012f4706-kube-api-access-nrwfk" pod "7abc0d07-ffde-4ca7-80d1-8461012f4706" (UID: "7abc0d07-ffde-4ca7-80d1-8461012f4706"). InnerVolumeSpecName "kube-api-access-nrwfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:17:51.472867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:51.472833    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrwfk\" (UniqueName: \"kubernetes.io/projected/7abc0d07-ffde-4ca7-80d1-8461012f4706-kube-api-access-nrwfk\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:17:51.472867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:51.472868    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:17:51.473076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:51.472882    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7abc0d07-ffde-4ca7-80d1-8461012f4706-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:17:52.124289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:52.124262    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"
Oct 30 21:17:52.124538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:52.124294    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk" event={"ID":"7abc0d07-ffde-4ca7-80d1-8461012f4706","Type":"ContainerDied","Data":"c04dbf7af73fc22cf6071080b807c1b2177986463b27c76ce1fdbd99a2561c24"}
Oct 30 21:17:52.124538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:52.124341    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c04dbf7af73fc22cf6071080b807c1b2177986463b27c76ce1fdbd99a2561c24"
Oct 30 21:17:52.863126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:52.863098    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"
Oct 30 21:17:52.863523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:17:52.863165    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"
Oct 30 21:17:59.235428 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:17:59.235242    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:18:01.016417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.012482    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"]
Oct 30 21:18:01.031752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.031691    4918 cpu_manager.go:414] "containerMap: removing container" podUID="7abc0d07-ffde-4ca7-80d1-8461012f4706" containerName="revoke-keys"
Oct 30 21:18:01.031987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.031966    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abc0d07-ffde-4ca7-80d1-8461012f4706" containerName="revoke-keys"
Oct 30 21:18:01.032425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.032406    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abc0d07-ffde-4ca7-80d1-8461012f4706" containerName="revoke-keys"
Oct 30 21:18:01.079890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.079853    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:01.135268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.135236    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:18:01.135469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.135288    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:18:01.135469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.135298    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:18:01.135469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.135351    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:18:01.142358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.142336    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"]
Oct 30 21:18:01.231184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.231152    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86671d-fmg77\" (UID: \"47f16e33-9ab6-4961-be65-b5016034a274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:01.231428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.231229    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmfrn\" (UniqueName: \"kubernetes.io/projected/47f16e33-9ab6-4961-be65-b5016034a274-kube-api-access-hmfrn\") pod \"maas-api-revoke-keys-e2e-api-a-86671d-fmg77\" (UID: \"47f16e33-9ab6-4961-be65-b5016034a274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:01.231497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.231477    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86671d-fmg77\" (UID: \"47f16e33-9ab6-4961-be65-b5016034a274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:01.332356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.332267    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86671d-fmg77\" (UID: \"47f16e33-9ab6-4961-be65-b5016034a274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:01.332539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.332429    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86671d-fmg77\" (UID: \"47f16e33-9ab6-4961-be65-b5016034a274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:01.332539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.332467    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmfrn\" (UniqueName: \"kubernetes.io/projected/47f16e33-9ab6-4961-be65-b5016034a274-kube-api-access-hmfrn\") pod \"maas-api-revoke-keys-e2e-api-a-86671d-fmg77\" (UID: \"47f16e33-9ab6-4961-be65-b5016034a274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:01.333176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.333090    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86671d-fmg77\" (UID: \"47f16e33-9ab6-4961-be65-b5016034a274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:01.334959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.334932    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86671d-fmg77\" (UID: \"47f16e33-9ab6-4961-be65-b5016034a274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:01.341321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.341301    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmfrn\" (UniqueName: \"kubernetes.io/projected/47f16e33-9ab6-4961-be65-b5016034a274-kube-api-access-hmfrn\") pod \"maas-api-revoke-keys-e2e-api-a-86671d-fmg77\" (UID: \"47f16e33-9ab6-4961-be65-b5016034a274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:01.449973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.449943    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:01.757941 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:18:01.757912    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f16e33_9ab6_4961_be65_b5016034a274.slice/crio-05871785a0264ef8c94512cf95863d45dd7d9d75a460a39432358c4fe82d06cc WatchSource:0}: Error finding container 05871785a0264ef8c94512cf95863d45dd7d9d75a460a39432358c4fe82d06cc: Status 404 returned error can't find the container with id 05871785a0264ef8c94512cf95863d45dd7d9d75a460a39432358c4fe82d06cc
Oct 30 21:18:01.759781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:01.759758    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"]
Oct 30 21:18:02.321691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:02.321663    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77" event={"ID":"47f16e33-9ab6-4961-be65-b5016034a274","Type":"ContainerStarted","Data":"05871785a0264ef8c94512cf95863d45dd7d9d75a460a39432358c4fe82d06cc"}
Oct 30 21:18:03.331459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:03.331341    4918 generic.go:356] "Generic (PLEG): container finished" podID="47f16e33-9ab6-4961-be65-b5016034a274" containerID="0fba8cd59e648977152949248d69653e3186db71875d3f55b499e71879ad03b8" exitCode=0
Oct 30 21:18:03.331459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:03.331440    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77" event={"ID":"47f16e33-9ab6-4961-be65-b5016034a274","Type":"ContainerDied","Data":"0fba8cd59e648977152949248d69653e3186db71875d3f55b499e71879ad03b8"}
Oct 30 21:18:04.223478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.223445    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"]
Oct 30 21:18:04.248204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.248159    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"]
Oct 30 21:18:04.248420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.248300    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:04.438300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.437879    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d789f639-j96ll\" (UID: \"b5b0d694-f94f-4036-afce-87a92ef93447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:04.438300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.437972    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7brvm\" (UniqueName: \"kubernetes.io/projected/b5b0d694-f94f-4036-afce-87a92ef93447-kube-api-access-7brvm\") pod \"maas-api-revoke-sub-d789f639-j96ll\" (UID: \"b5b0d694-f94f-4036-afce-87a92ef93447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:04.438300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.438011    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d789f639-j96ll\" (UID: \"b5b0d694-f94f-4036-afce-87a92ef93447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:04.542039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.532997    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:04.543269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.543240    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7brvm\" (UniqueName: \"kubernetes.io/projected/b5b0d694-f94f-4036-afce-87a92ef93447-kube-api-access-7brvm\") pod \"maas-api-revoke-sub-d789f639-j96ll\" (UID: \"b5b0d694-f94f-4036-afce-87a92ef93447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:04.543422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.543278    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d789f639-j96ll\" (UID: \"b5b0d694-f94f-4036-afce-87a92ef93447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:04.543525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.543511    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d789f639-j96ll\" (UID: \"b5b0d694-f94f-4036-afce-87a92ef93447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:04.544088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.544067    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d789f639-j96ll\" (UID: \"b5b0d694-f94f-4036-afce-87a92ef93447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:04.545992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.545970    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d789f639-j96ll\" (UID: \"b5b0d694-f94f-4036-afce-87a92ef93447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:04.552237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.552215    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7brvm\" (UniqueName: \"kubernetes.io/projected/b5b0d694-f94f-4036-afce-87a92ef93447-kube-api-access-7brvm\") pod \"maas-api-revoke-sub-d789f639-j96ll\" (UID: \"b5b0d694-f94f-4036-afce-87a92ef93447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:04.565975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.565953    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:04.645281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.645231    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-xks-ca\") pod \"47f16e33-9ab6-4961-be65-b5016034a274\" (UID: \"47f16e33-9ab6-4961-be65-b5016034a274\") "
Oct 30 21:18:04.645538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.645514    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-openshift-ca\") pod \"47f16e33-9ab6-4961-be65-b5016034a274\" (UID: \"47f16e33-9ab6-4961-be65-b5016034a274\") "
Oct 30 21:18:04.645691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.645619    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f16e33-9ab6-4961-be65-b5016034a274-kube-api-access-hmfrn\" (UniqueName: \"kubernetes.io/projected/47f16e33-9ab6-4961-be65-b5016034a274-kube-api-access-hmfrn\") pod \"47f16e33-9ab6-4961-be65-b5016034a274\" (UID: \"47f16e33-9ab6-4961-be65-b5016034a274\") "
Oct 30 21:18:04.646365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.646335    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-openshift-ca" pod "47f16e33-9ab6-4961-be65-b5016034a274" (UID: "47f16e33-9ab6-4961-be65-b5016034a274"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:04.648443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.648419    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-xks-ca" pod "47f16e33-9ab6-4961-be65-b5016034a274" (UID: "47f16e33-9ab6-4961-be65-b5016034a274"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:04.648524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.648460    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f16e33-9ab6-4961-be65-b5016034a274-kube-api-access-hmfrn" pod "47f16e33-9ab6-4961-be65-b5016034a274" (UID: "47f16e33-9ab6-4961-be65-b5016034a274"). InnerVolumeSpecName "kube-api-access-hmfrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:04.748124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.748050    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:04.748303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.748288    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f16e33-9ab6-4961-be65-b5016034a274-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:04.748381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.748373    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmfrn\" (UniqueName: \"kubernetes.io/projected/47f16e33-9ab6-4961-be65-b5016034a274-kube-api-access-hmfrn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:04.930158 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:04.928881    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"]
Oct 30 21:18:05.351130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:05.351029    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll" event={"ID":"b5b0d694-f94f-4036-afce-87a92ef93447","Type":"ContainerStarted","Data":"bf5da1143da4fe56bab4d762e96bf713386f748d2b1e17be3a819da80249f666"}
Oct 30 21:18:05.351130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:05.351099    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll" event={"ID":"b5b0d694-f94f-4036-afce-87a92ef93447","Type":"ContainerStarted","Data":"98b529212f6bb963cecd76b965c21385b32da3cc6f862c8d349e647dce0ab196"}
Oct 30 21:18:05.352659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:05.352641    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"
Oct 30 21:18:05.352794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:05.352640    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77" event={"ID":"47f16e33-9ab6-4961-be65-b5016034a274","Type":"ContainerDied","Data":"05871785a0264ef8c94512cf95863d45dd7d9d75a460a39432358c4fe82d06cc"}
Oct 30 21:18:05.352794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:05.352749    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05871785a0264ef8c94512cf95863d45dd7d9d75a460a39432358c4fe82d06cc"
Oct 30 21:18:05.375232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:05.375186    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll" podStartSLOduration=1.375173612 podStartE2EDuration="1.375173612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:05.369079116 +0000 UTC m=+998.415638301" watchObservedRunningTime="2026-10-30 21:18:05.375173612 +0000 UTC m=+998.421732751"
Oct 30 21:18:06.414438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:06.414372    4918 generic.go:356] "Generic (PLEG): container finished" podID="b5b0d694-f94f-4036-afce-87a92ef93447" containerID="bf5da1143da4fe56bab4d762e96bf713386f748d2b1e17be3a819da80249f666" exitCode=0
Oct 30 21:18:06.414958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:06.414701    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll" event={"ID":"b5b0d694-f94f-4036-afce-87a92ef93447","Type":"ContainerDied","Data":"bf5da1143da4fe56bab4d762e96bf713386f748d2b1e17be3a819da80249f666"}
Oct 30 21:18:07.270498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:07.270460    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"]
Oct 30 21:18:07.297416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:07.295359    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6a24e1-cqrvk"]
Oct 30 21:18:07.665203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:07.665175    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:07.744726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:07.744692    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-openshift-ca\") pod \"b5b0d694-f94f-4036-afce-87a92ef93447\" (UID: \"b5b0d694-f94f-4036-afce-87a92ef93447\") "
Oct 30 21:18:07.745024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:07.744764    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b0d694-f94f-4036-afce-87a92ef93447-kube-api-access-7brvm\" (UniqueName: \"kubernetes.io/projected/b5b0d694-f94f-4036-afce-87a92ef93447-kube-api-access-7brvm\") pod \"b5b0d694-f94f-4036-afce-87a92ef93447\" (UID: \"b5b0d694-f94f-4036-afce-87a92ef93447\") "
Oct 30 21:18:07.745024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:07.745008    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-xks-ca\") pod \"b5b0d694-f94f-4036-afce-87a92ef93447\" (UID: \"b5b0d694-f94f-4036-afce-87a92ef93447\") "
Oct 30 21:18:07.746416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:07.746371    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-openshift-ca" pod "b5b0d694-f94f-4036-afce-87a92ef93447" (UID: "b5b0d694-f94f-4036-afce-87a92ef93447"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:07.748620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:07.748598    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-xks-ca" pod "b5b0d694-f94f-4036-afce-87a92ef93447" (UID: "b5b0d694-f94f-4036-afce-87a92ef93447"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:07.748929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:07.748908    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b0d694-f94f-4036-afce-87a92ef93447-kube-api-access-7brvm" pod "b5b0d694-f94f-4036-afce-87a92ef93447" (UID: "b5b0d694-f94f-4036-afce-87a92ef93447"). InnerVolumeSpecName "kube-api-access-7brvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:07.847484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:07.847451    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:07.847484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:07.847484    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5b0d694-f94f-4036-afce-87a92ef93447-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:07.847712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:07.847499    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7brvm\" (UniqueName: \"kubernetes.io/projected/b5b0d694-f94f-4036-afce-87a92ef93447-kube-api-access-7brvm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:08.434606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:08.434575    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll" event={"ID":"b5b0d694-f94f-4036-afce-87a92ef93447","Type":"ContainerDied","Data":"98b529212f6bb963cecd76b965c21385b32da3cc6f862c8d349e647dce0ab196"}
Oct 30 21:18:08.434606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:08.434598    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"
Oct 30 21:18:08.434811 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:08.434622    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98b529212f6bb963cecd76b965c21385b32da3cc6f862c8d349e647dce0ab196"
Oct 30 21:18:09.079682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:09.079644    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7abc0d07-ffde-4ca7-80d1-8461012f4706" path="/var/lib/kubelet/pods/7abc0d07-ffde-4ca7-80d1-8461012f4706/volumes"
Oct 30 21:18:09.259572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:09.257615    4918 patch_prober.go:29] interesting pod/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": read tcp 10.132.0.2:49544->10.132.0.126:15021: read: connection reset by peer" start-of-body=
Oct 30 21:18:09.259572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:09.257674    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": read tcp 10.132.0.2:49544->10.132.0.126:15021: read: connection reset by peer"
Oct 30 21:18:09.444738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:09.444635    4918 generic.go:356] "Generic (PLEG): container finished" podID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerID="5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec" exitCode=0
Oct 30 21:18:09.444738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:09.444710    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" event={"ID":"6ae7a5a7-5cd7-4094-863c-2de252eda90b","Type":"ContainerDied","Data":"5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec"}
Oct 30 21:18:09.445142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:09.445130    4918 scope.go:122] "RemoveContainer" containerID="5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec"
Oct 30 21:18:09.446754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:09.446719    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:18:09.446835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:09.446783    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:18:09.446835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:09.446817    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:18:10.040656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:10.040630    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"]
Oct 30 21:18:10.045649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:10.045624    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"]
Oct 30 21:18:10.050122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:10.050103    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb2fa007-dphkr"]
Oct 30 21:18:10.054052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:10.054030    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d76c6f06-z7b4z"]
Oct 30 21:18:10.458661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:10.458572    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" event={"ID":"6ae7a5a7-5cd7-4094-863c-2de252eda90b","Type":"ContainerStarted","Data":"30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d"}
Oct 30 21:18:10.675864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:10.675833    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:18:10.676035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:10.675966    4918 patch_prober.go:29] interesting pod/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:10.676035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:10.676014    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 21:18:11.082269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:11.082237    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5a8708-252d-4451-839f-49802615aa93" path="/var/lib/kubelet/pods/4c5a8708-252d-4451-839f-49802615aa93/volumes"
Oct 30 21:18:11.082766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:11.082752    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cf7a0e-4830-4d88-9bf7-a1774d8a662b" path="/var/lib/kubelet/pods/b8cf7a0e-4830-4d88-9bf7-a1774d8a662b/volumes"
Oct 30 21:18:11.182449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:11.182417    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"]
Oct 30 21:18:11.680418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:11.678760    4918 patch_prober.go:29] interesting pod/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:11.680418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:11.678817    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 21:18:12.675971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:12.675940    4918 patch_prober.go:29] interesting pod/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:12.676153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:12.675986    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 21:18:13.675946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:13.675913    4918 patch_prober.go:29] interesting pod/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:13.676350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:13.675960    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 21:18:14.679414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:14.677609    4918 patch_prober.go:29] interesting pod/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:14.679414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:14.677666    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 21:18:14.929470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:14.925188    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"]
Oct 30 21:18:14.929470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:14.926215    4918 cpu_manager.go:414] "containerMap: removing container" podUID="47f16e33-9ab6-4961-be65-b5016034a274" containerName="revoke-keys"
Oct 30 21:18:14.929470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:14.926236    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f16e33-9ab6-4961-be65-b5016034a274" containerName="revoke-keys"
Oct 30 21:18:14.929470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:14.926281    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b5b0d694-f94f-4036-afce-87a92ef93447" containerName="revoke-keys"
Oct 30 21:18:14.929470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:14.926288    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b0d694-f94f-4036-afce-87a92ef93447" containerName="revoke-keys"
Oct 30 21:18:14.929470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:14.926519    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f16e33-9ab6-4961-be65-b5016034a274" containerName="revoke-keys"
Oct 30 21:18:14.929470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:14.926541    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b0d694-f94f-4036-afce-87a92ef93447" containerName="revoke-keys"
Oct 30 21:18:15.017410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.016492    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"]
Oct 30 21:18:15.017410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.016622    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:15.043005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.041980    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:18:15.043533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.043514    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:18:15.049670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.048727    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:18:15.049670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.048793    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:18:15.055492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.054216    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w6ts\" (UniqueName: \"kubernetes.io/projected/ae7db77f-68bb-4b4a-8cbe-66beee304169-kube-api-access-4w6ts\") pod \"maas-api-revoke-keys-e2e-api-b-c499bf-knb2t\" (UID: \"ae7db77f-68bb-4b4a-8cbe-66beee304169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:15.055492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.054265    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c499bf-knb2t\" (UID: \"ae7db77f-68bb-4b4a-8cbe-66beee304169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:15.055492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.054304    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c499bf-knb2t\" (UID: \"ae7db77f-68bb-4b4a-8cbe-66beee304169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:15.156115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.156083    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w6ts\" (UniqueName: \"kubernetes.io/projected/ae7db77f-68bb-4b4a-8cbe-66beee304169-kube-api-access-4w6ts\") pod \"maas-api-revoke-keys-e2e-api-b-c499bf-knb2t\" (UID: \"ae7db77f-68bb-4b4a-8cbe-66beee304169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:15.156484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.156122    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c499bf-knb2t\" (UID: \"ae7db77f-68bb-4b4a-8cbe-66beee304169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:15.156484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.156159    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c499bf-knb2t\" (UID: \"ae7db77f-68bb-4b4a-8cbe-66beee304169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:15.156847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.156823    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c499bf-knb2t\" (UID: \"ae7db77f-68bb-4b4a-8cbe-66beee304169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:15.159715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.159685    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c499bf-knb2t\" (UID: \"ae7db77f-68bb-4b4a-8cbe-66beee304169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:15.164684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.164665    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w6ts\" (UniqueName: \"kubernetes.io/projected/ae7db77f-68bb-4b4a-8cbe-66beee304169-kube-api-access-4w6ts\") pod \"maas-api-revoke-keys-e2e-api-b-c499bf-knb2t\" (UID: \"ae7db77f-68bb-4b4a-8cbe-66beee304169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:15.359878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.359849    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:15.683505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.679217    4918 patch_prober.go:29] interesting pod/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 21:18:15.683505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.679295    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 21:18:15.700420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:15.696040    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"]
Oct 30 21:18:15.723430 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:18:15.720502    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7db77f_68bb_4b4a_8cbe_66beee304169.slice/crio-833be14e108920301140c81c434cc873b6cbad346e35fa755eac8cd2a017f8ba WatchSource:0}: Error finding container 833be14e108920301140c81c434cc873b6cbad346e35fa755eac8cd2a017f8ba: Status 404 returned error can't find the container with id 833be14e108920301140c81c434cc873b6cbad346e35fa755eac8cd2a017f8ba
Oct 30 21:18:16.182903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.182865    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8" podUID="cea25e13-be53-4cce-81bf-2ff2aa342f3b" containerName="payload-processing" containerID="cri-o://074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d" gracePeriod=25
Oct 30 21:18:16.569692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.569661    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"
Oct 30 21:18:16.576514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.572867    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t" event={"ID":"ae7db77f-68bb-4b4a-8cbe-66beee304169","Type":"ContainerStarted","Data":"b73fbe2a2ee65cef71c0c2fabbbe59e7d5b0fca107154c3763e2b990e74a5cbe"}
Oct 30 21:18:16.576514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.572936    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t" event={"ID":"ae7db77f-68bb-4b4a-8cbe-66beee304169","Type":"ContainerStarted","Data":"833be14e108920301140c81c434cc873b6cbad346e35fa755eac8cd2a017f8ba"}
Oct 30 21:18:16.576514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.574524    4918 generic.go:356] "Generic (PLEG): container finished" podID="cea25e13-be53-4cce-81bf-2ff2aa342f3b" containerID="074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d" exitCode=0
Oct 30 21:18:16.576514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.574625    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8" event={"ID":"cea25e13-be53-4cce-81bf-2ff2aa342f3b","Type":"ContainerDied","Data":"074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d"}
Oct 30 21:18:16.576514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.574661    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8" event={"ID":"cea25e13-be53-4cce-81bf-2ff2aa342f3b","Type":"ContainerDied","Data":"be51be62b5bc65edae66ccd3c7b8b0d111d9cd2e93ff2e79a5d866d5db6d3659"}
Oct 30 21:18:16.576514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.574684    4918 scope.go:122] "RemoveContainer" containerID="074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d"
Oct 30 21:18:16.576514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.574795    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"
Oct 30 21:18:16.585553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.585528    4918 scope.go:122] "RemoveContainer" containerID="074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d"
Oct 30 21:18:16.585870 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:18:16.585846    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d\": container with ID starting with 074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d not found: ID does not exist" containerID="074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d"
Oct 30 21:18:16.585985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.585876    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d"} err="failed to get container status \"074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d\": rpc error: code = NotFound desc = could not find container \"074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d\": container with ID starting with 074fc8779652a202cd00d2c26e525ba5b8f74025ae19460e14e8b0e6a94aa73d not found: ID does not exist"
Oct 30 21:18:16.607086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.607037    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t" podStartSLOduration=2.607016819 podStartE2EDuration="2.607016819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:16.602798986 +0000 UTC m=+1009.649358152" watchObservedRunningTime="2026-10-30 21:18:16.607016819 +0000 UTC m=+1009.653575964"
Oct 30 21:18:16.623574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.623544    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea25e13-be53-4cce-81bf-2ff2aa342f3b-kube-api-access-6dls5\" (UniqueName: \"kubernetes.io/projected/cea25e13-be53-4cce-81bf-2ff2aa342f3b-kube-api-access-6dls5\") pod \"cea25e13-be53-4cce-81bf-2ff2aa342f3b\" (UID: \"cea25e13-be53-4cce-81bf-2ff2aa342f3b\") "
Oct 30 21:18:16.623781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.623647    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea25e13-be53-4cce-81bf-2ff2aa342f3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea25e13-be53-4cce-81bf-2ff2aa342f3b-plugins-config-volume\") pod \"cea25e13-be53-4cce-81bf-2ff2aa342f3b\" (UID: \"cea25e13-be53-4cce-81bf-2ff2aa342f3b\") "
Oct 30 21:18:16.624110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.624086    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea25e13-be53-4cce-81bf-2ff2aa342f3b-plugins-config-volume" pod "cea25e13-be53-4cce-81bf-2ff2aa342f3b" (UID: "cea25e13-be53-4cce-81bf-2ff2aa342f3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:16.624575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.624548    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea25e13-be53-4cce-81bf-2ff2aa342f3b-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:16.642566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.642538    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea25e13-be53-4cce-81bf-2ff2aa342f3b-kube-api-access-6dls5" pod "cea25e13-be53-4cce-81bf-2ff2aa342f3b" (UID: "cea25e13-be53-4cce-81bf-2ff2aa342f3b"). InnerVolumeSpecName "kube-api-access-6dls5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:16.680508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.680484    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:18:16.680954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.680925    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:18:16.681663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.681642    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:18:16.726007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.725982    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dls5\" (UniqueName: \"kubernetes.io/projected/cea25e13-be53-4cce-81bf-2ff2aa342f3b-kube-api-access-6dls5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:16.901229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.901201    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"]
Oct 30 21:18:16.905762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:16.905741    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-86671d-57c7476599-fwsr8"]
Oct 30 21:18:17.144669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:17.144571    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea25e13-be53-4cce-81bf-2ff2aa342f3b" path="/var/lib/kubelet/pods/cea25e13-be53-4cce-81bf-2ff2aa342f3b/volumes"
Oct 30 21:18:17.585217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:17.585180    4918 generic.go:356] "Generic (PLEG): container finished" podID="ae7db77f-68bb-4b4a-8cbe-66beee304169" containerID="b73fbe2a2ee65cef71c0c2fabbbe59e7d5b0fca107154c3763e2b990e74a5cbe" exitCode=0
Oct 30 21:18:17.585429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:17.585256    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t" event={"ID":"ae7db77f-68bb-4b4a-8cbe-66beee304169","Type":"ContainerDied","Data":"b73fbe2a2ee65cef71c0c2fabbbe59e7d5b0fca107154c3763e2b990e74a5cbe"}
Oct 30 21:18:18.841482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:18.841458    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:18.973373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:18.973343    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7db77f-68bb-4b4a-8cbe-66beee304169-kube-api-access-4w6ts\" (UniqueName: \"kubernetes.io/projected/ae7db77f-68bb-4b4a-8cbe-66beee304169-kube-api-access-4w6ts\") pod \"ae7db77f-68bb-4b4a-8cbe-66beee304169\" (UID: \"ae7db77f-68bb-4b4a-8cbe-66beee304169\") "
Oct 30 21:18:18.973580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:18.973470    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-xks-ca\") pod \"ae7db77f-68bb-4b4a-8cbe-66beee304169\" (UID: \"ae7db77f-68bb-4b4a-8cbe-66beee304169\") "
Oct 30 21:18:18.973627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:18.973582    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-openshift-ca\") pod \"ae7db77f-68bb-4b4a-8cbe-66beee304169\" (UID: \"ae7db77f-68bb-4b4a-8cbe-66beee304169\") "
Oct 30 21:18:18.973933 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:18.973909    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-openshift-ca" pod "ae7db77f-68bb-4b4a-8cbe-66beee304169" (UID: "ae7db77f-68bb-4b4a-8cbe-66beee304169"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:18.974248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:18.974180    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:18.975725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:18.975700    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-xks-ca" pod "ae7db77f-68bb-4b4a-8cbe-66beee304169" (UID: "ae7db77f-68bb-4b4a-8cbe-66beee304169"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:18.975804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:18.975780    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7db77f-68bb-4b4a-8cbe-66beee304169-kube-api-access-4w6ts" pod "ae7db77f-68bb-4b4a-8cbe-66beee304169" (UID: "ae7db77f-68bb-4b4a-8cbe-66beee304169"). InnerVolumeSpecName "kube-api-access-4w6ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:19.026457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:19.026426    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"]
Oct 30 21:18:19.031081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:19.031053    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7486dadd-x5qr5"]
Oct 30 21:18:19.075460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:19.075432    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w6ts\" (UniqueName: \"kubernetes.io/projected/ae7db77f-68bb-4b4a-8cbe-66beee304169-kube-api-access-4w6ts\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:19.075460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:19.075456    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae7db77f-68bb-4b4a-8cbe-66beee304169-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:19.081481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:19.081455    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af55126d-0189-44ab-892b-4b363d3738cd" path="/var/lib/kubelet/pods/af55126d-0189-44ab-892b-4b363d3738cd/volumes"
Oct 30 21:18:19.638193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:19.638158    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t" event={"ID":"ae7db77f-68bb-4b4a-8cbe-66beee304169","Type":"ContainerDied","Data":"833be14e108920301140c81c434cc873b6cbad346e35fa755eac8cd2a017f8ba"}
Oct 30 21:18:19.638483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:19.638465    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="833be14e108920301140c81c434cc873b6cbad346e35fa755eac8cd2a017f8ba"
Oct 30 21:18:19.638659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:19.638647    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"
Oct 30 21:18:25.026531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.026494    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"]
Oct 30 21:18:25.031322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.031298    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38a5f909-c96w7"]
Oct 30 21:18:25.035152 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.035134    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"]
Oct 30 21:18:25.038768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.038747    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-191380e4-fvbj2"]
Oct 30 21:18:25.054551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.054526    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"]
Oct 30 21:18:25.064563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.064535    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"]
Oct 30 21:18:25.090179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.089183    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a11411-a922-4fd1-82c6-0f34f3dd2500" path="/var/lib/kubelet/pods/93a11411-a922-4fd1-82c6-0f34f3dd2500/volumes"
Oct 30 21:18:25.090179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.089916    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2f1586-4ce8-4898-a3be-06b6fd562a33" path="/var/lib/kubelet/pods/ce2f1586-4ce8-4898-a3be-06b6fd562a33/volumes"
Oct 30 21:18:25.941534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.941499    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"]
Oct 30 21:18:25.942157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.942143    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ae7db77f-68bb-4b4a-8cbe-66beee304169" containerName="revoke-keys"
Oct 30 21:18:25.942203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.942160    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7db77f-68bb-4b4a-8cbe-66beee304169" containerName="revoke-keys"
Oct 30 21:18:25.942203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.942199    4918 cpu_manager.go:414] "containerMap: removing container" podUID="cea25e13-be53-4cce-81bf-2ff2aa342f3b" containerName="payload-processing"
Oct 30 21:18:25.942261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.942205    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea25e13-be53-4cce-81bf-2ff2aa342f3b" containerName="payload-processing"
Oct 30 21:18:25.942305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.942297    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea25e13-be53-4cce-81bf-2ff2aa342f3b" containerName="payload-processing"
Oct 30 21:18:25.942342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.942311    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7db77f-68bb-4b4a-8cbe-66beee304169" containerName="revoke-keys"
Oct 30 21:18:25.951899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.947484    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:25.952082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.952043    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:18:25.955414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.953175    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:18:25.955414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.953264    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:18:25.955414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.953424    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:18:25.955637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:25.955510    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"]
Oct 30 21:18:26.036977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:26.036934    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5jxp\" (UniqueName: \"kubernetes.io/projected/fe0609cc-463d-401e-9117-754584f74c9a-kube-api-access-m5jxp\") pod \"maas-api-revoke-sub-91cbf037-7dddv\" (UID: \"fe0609cc-463d-401e-9117-754584f74c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:26.037342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:26.037123    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91cbf037-7dddv\" (UID: \"fe0609cc-463d-401e-9117-754584f74c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:26.037342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:26.037154    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91cbf037-7dddv\" (UID: \"fe0609cc-463d-401e-9117-754584f74c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:26.138756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:26.138719    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91cbf037-7dddv\" (UID: \"fe0609cc-463d-401e-9117-754584f74c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:26.138918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:26.138779    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91cbf037-7dddv\" (UID: \"fe0609cc-463d-401e-9117-754584f74c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:26.138991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:26.138959    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5jxp\" (UniqueName: \"kubernetes.io/projected/fe0609cc-463d-401e-9117-754584f74c9a-kube-api-access-m5jxp\") pod \"maas-api-revoke-sub-91cbf037-7dddv\" (UID: \"fe0609cc-463d-401e-9117-754584f74c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:26.139790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:26.139759    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91cbf037-7dddv\" (UID: \"fe0609cc-463d-401e-9117-754584f74c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:26.141646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:26.141621    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91cbf037-7dddv\" (UID: \"fe0609cc-463d-401e-9117-754584f74c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:26.147911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:26.147885    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5jxp\" (UniqueName: \"kubernetes.io/projected/fe0609cc-463d-401e-9117-754584f74c9a-kube-api-access-m5jxp\") pod \"maas-api-revoke-sub-91cbf037-7dddv\" (UID: \"fe0609cc-463d-401e-9117-754584f74c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:26.279739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:26.279708    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:26.658522 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:18:26.658468    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0609cc_463d_401e_9117_754584f74c9a.slice/crio-fdcf7c8b7cd5071a49b27169f12c7416915478e928ecd1dcedb00b6b3689aca4 WatchSource:0}: Error finding container fdcf7c8b7cd5071a49b27169f12c7416915478e928ecd1dcedb00b6b3689aca4: Status 404 returned error can't find the container with id fdcf7c8b7cd5071a49b27169f12c7416915478e928ecd1dcedb00b6b3689aca4
Oct 30 21:18:26.659947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:26.659922    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"]
Oct 30 21:18:26.717400 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:26.717359    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv" event={"ID":"fe0609cc-463d-401e-9117-754584f74c9a","Type":"ContainerStarted","Data":"fdcf7c8b7cd5071a49b27169f12c7416915478e928ecd1dcedb00b6b3689aca4"}
Oct 30 21:18:27.726459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:27.726421    4918 generic.go:356] "Generic (PLEG): container finished" podID="fe0609cc-463d-401e-9117-754584f74c9a" containerID="26513dba28d48897afbc705c47653e37f4d0eaef640f296d6b6156e7ef4efc8e" exitCode=0
Oct 30 21:18:27.726865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:27.726525    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv" event={"ID":"fe0609cc-463d-401e-9117-754584f74c9a","Type":"ContainerDied","Data":"26513dba28d48897afbc705c47653e37f4d0eaef640f296d6b6156e7ef4efc8e"}
Oct 30 21:18:28.456755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:28.456719    4918 scope.go:122] "RemoveContainer" containerID="11d4fd0f5d627048687c4d621ffe082c9fb129e5fb2c40495f92d773b080ee33"
Oct 30 21:18:28.467212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:28.467189    4918 scope.go:122] "RemoveContainer" containerID="409570eb85ef697b7a928b8c2f0264f17baec804ce69d2cde6fa3722ba0ddcf0"
Oct 30 21:18:28.476707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:28.476688    4918 scope.go:122] "RemoveContainer" containerID="0f22c44141db262777a5db6f20c9da19abb2822938bf23b06bbf4dd1a764d352"
Oct 30 21:18:28.486165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:28.486142    4918 scope.go:122] "RemoveContainer" containerID="eda9a3b324c9ec7a5f26070f9cf98c2c629d5ee61bf7001dc235495297eba85b"
Oct 30 21:18:28.806424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:28.795853    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"]
Oct 30 21:18:28.806424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:28.803525    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86671d-fmg77"]
Oct 30 21:18:29.071621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.071598    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:29.080297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.080271    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f16e33-9ab6-4961-be65-b5016034a274" path="/var/lib/kubelet/pods/47f16e33-9ab6-4961-be65-b5016034a274/volumes"
Oct 30 21:18:29.098264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.098235    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0609cc-463d-401e-9117-754584f74c9a-kube-api-access-m5jxp\" (UniqueName: \"kubernetes.io/projected/fe0609cc-463d-401e-9117-754584f74c9a-kube-api-access-m5jxp\") pod \"fe0609cc-463d-401e-9117-754584f74c9a\" (UID: \"fe0609cc-463d-401e-9117-754584f74c9a\") "
Oct 30 21:18:29.098413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.098324    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-xks-ca\") pod \"fe0609cc-463d-401e-9117-754584f74c9a\" (UID: \"fe0609cc-463d-401e-9117-754584f74c9a\") "
Oct 30 21:18:29.098511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.098468    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-openshift-ca\") pod \"fe0609cc-463d-401e-9117-754584f74c9a\" (UID: \"fe0609cc-463d-401e-9117-754584f74c9a\") "
Oct 30 21:18:29.099209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.098812    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-openshift-ca" pod "fe0609cc-463d-401e-9117-754584f74c9a" (UID: "fe0609cc-463d-401e-9117-754584f74c9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:29.099209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.099115    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:29.100761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.100733    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0609cc-463d-401e-9117-754584f74c9a-kube-api-access-m5jxp" pod "fe0609cc-463d-401e-9117-754584f74c9a" (UID: "fe0609cc-463d-401e-9117-754584f74c9a"). InnerVolumeSpecName "kube-api-access-m5jxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:29.101107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.101083    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-xks-ca" pod "fe0609cc-463d-401e-9117-754584f74c9a" (UID: "fe0609cc-463d-401e-9117-754584f74c9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:29.167512 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:18:29.167468    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:18:29.201156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.201123    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5jxp\" (UniqueName: \"kubernetes.io/projected/fe0609cc-463d-401e-9117-754584f74c9a-kube-api-access-m5jxp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:29.201156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.201160    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0609cc-463d-401e-9117-754584f74c9a-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:29.772507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.772475    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"
Oct 30 21:18:29.772507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.772489    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv" event={"ID":"fe0609cc-463d-401e-9117-754584f74c9a","Type":"ContainerDied","Data":"fdcf7c8b7cd5071a49b27169f12c7416915478e928ecd1dcedb00b6b3689aca4"}
Oct 30 21:18:29.772804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:29.772538    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdcf7c8b7cd5071a49b27169f12c7416915478e928ecd1dcedb00b6b3689aca4"
Oct 30 21:18:30.054958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.054876    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9" podUID="bbff0160-e9cf-4b95-848d-0336168b038c" containerName="payload-processing" containerID="cri-o://d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984" gracePeriod=25
Oct 30 21:18:30.065425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.065377    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b" podUID="acc8d8d8-6d8c-4c65-adb2-413fa37ee354" containerName="payload-pre-processing" containerID="cri-o://f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f" gracePeriod=25
Oct 30 21:18:30.425413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.412587    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"
Oct 30 21:18:30.425413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.420870    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbff0160-e9cf-4b95-848d-0336168b038c-kube-api-access-cshfd\" (UniqueName: \"kubernetes.io/projected/bbff0160-e9cf-4b95-848d-0336168b038c-kube-api-access-cshfd\") pod \"bbff0160-e9cf-4b95-848d-0336168b038c\" (UID: \"bbff0160-e9cf-4b95-848d-0336168b038c\") "
Oct 30 21:18:30.425413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.421018    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbff0160-e9cf-4b95-848d-0336168b038c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbff0160-e9cf-4b95-848d-0336168b038c-plugins-config-volume\") pod \"bbff0160-e9cf-4b95-848d-0336168b038c\" (UID: \"bbff0160-e9cf-4b95-848d-0336168b038c\") "
Oct 30 21:18:30.425413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.422599    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbff0160-e9cf-4b95-848d-0336168b038c-plugins-config-volume" pod "bbff0160-e9cf-4b95-848d-0336168b038c" (UID: "bbff0160-e9cf-4b95-848d-0336168b038c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:30.425413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.425139    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbff0160-e9cf-4b95-848d-0336168b038c-kube-api-access-cshfd" pod "bbff0160-e9cf-4b95-848d-0336168b038c" (UID: "bbff0160-e9cf-4b95-848d-0336168b038c"). InnerVolumeSpecName "kube-api-access-cshfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:30.490760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.490109    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"
Oct 30 21:18:30.523145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.523118    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-kube-api-access-pdwjj\" (UniqueName: \"kubernetes.io/projected/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-kube-api-access-pdwjj\") pod \"acc8d8d8-6d8c-4c65-adb2-413fa37ee354\" (UID: \"acc8d8d8-6d8c-4c65-adb2-413fa37ee354\") "
Oct 30 21:18:30.523304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.523256    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-plugins-config-volume\") pod \"acc8d8d8-6d8c-4c65-adb2-413fa37ee354\" (UID: \"acc8d8d8-6d8c-4c65-adb2-413fa37ee354\") "
Oct 30 21:18:30.523761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.523732    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-plugins-config-volume" pod "acc8d8d8-6d8c-4c65-adb2-413fa37ee354" (UID: "acc8d8d8-6d8c-4c65-adb2-413fa37ee354"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:30.524114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.524095    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbff0160-e9cf-4b95-848d-0336168b038c-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:30.524201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.524119    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:30.524201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.524133    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cshfd\" (UniqueName: \"kubernetes.io/projected/bbff0160-e9cf-4b95-848d-0336168b038c-kube-api-access-cshfd\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:30.525619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.525586    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-kube-api-access-pdwjj" pod "acc8d8d8-6d8c-4c65-adb2-413fa37ee354" (UID: "acc8d8d8-6d8c-4c65-adb2-413fa37ee354"). InnerVolumeSpecName "kube-api-access-pdwjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:30.625953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.625874    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdwjj\" (UniqueName: \"kubernetes.io/projected/acc8d8d8-6d8c-4c65-adb2-413fa37ee354-kube-api-access-pdwjj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:30.781744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.781712    4918 generic.go:356] "Generic (PLEG): container finished" podID="acc8d8d8-6d8c-4c65-adb2-413fa37ee354" containerID="f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f" exitCode=0
Oct 30 21:18:30.781915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.781755    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"
Oct 30 21:18:30.781915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.781792    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b" event={"ID":"acc8d8d8-6d8c-4c65-adb2-413fa37ee354","Type":"ContainerDied","Data":"f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f"}
Oct 30 21:18:30.781915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.781869    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b" event={"ID":"acc8d8d8-6d8c-4c65-adb2-413fa37ee354","Type":"ContainerDied","Data":"4e9a59238fad32c702b191d8ad63ec79890d5b6afea6eab3d442756ce4c35bd9"}
Oct 30 21:18:30.781915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.781882    4918 scope.go:122] "RemoveContainer" containerID="f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f"
Oct 30 21:18:30.783554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.783523    4918 generic.go:356] "Generic (PLEG): container finished" podID="bbff0160-e9cf-4b95-848d-0336168b038c" containerID="d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984" exitCode=0
Oct 30 21:18:30.783662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.783573    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"
Oct 30 21:18:30.783717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.783654    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9" event={"ID":"bbff0160-e9cf-4b95-848d-0336168b038c","Type":"ContainerDied","Data":"d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984"}
Oct 30 21:18:30.783717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.783707    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9" event={"ID":"bbff0160-e9cf-4b95-848d-0336168b038c","Type":"ContainerDied","Data":"3cf05e9568aeaa076ffda237cf1aae5daf8896799a0809b4414702de6b1a2789"}
Oct 30 21:18:30.794288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.794272    4918 scope.go:122] "RemoveContainer" containerID="f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f"
Oct 30 21:18:30.794606 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:18:30.794580    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f\": container with ID starting with f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f not found: ID does not exist" containerID="f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f"
Oct 30 21:18:30.794719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.794616    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f"} err="failed to get container status \"f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f\": rpc error: code = NotFound desc = could not find container \"f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f\": container with ID starting with f11fecaa438c5383bd071fdd48b35888a9f1d8b519bba9baf3b737bc5e9cdb6f not found: ID does not exist"
Oct 30 21:18:30.794719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.794636    4918 scope.go:122] "RemoveContainer" containerID="d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984"
Oct 30 21:18:30.804920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.804900    4918 scope.go:122] "RemoveContainer" containerID="d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984"
Oct 30 21:18:30.805246 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:18:30.805215    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984\": container with ID starting with d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984 not found: ID does not exist" containerID="d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984"
Oct 30 21:18:30.805304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.805254    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984"} err="failed to get container status \"d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984\": rpc error: code = NotFound desc = could not find container \"d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984\": container with ID starting with d89f89019fdb8299befdd57c30289f7c4b6117281ab6482f300e002811e03984 not found: ID does not exist"
Oct 30 21:18:30.812606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.812585    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"]
Oct 30 21:18:30.816860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.816839    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9"]
Oct 30 21:18:30.825681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.825654    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"]
Oct 30 21:18:30.830100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:30.830080    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b"]
Oct 30 21:18:31.143654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:31.143612    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc8d8d8-6d8c-4c65-adb2-413fa37ee354" path="/var/lib/kubelet/pods/acc8d8d8-6d8c-4c65-adb2-413fa37ee354/volumes"
Oct 30 21:18:31.144263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:31.144240    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbff0160-e9cf-4b95-848d-0336168b038c" path="/var/lib/kubelet/pods/bbff0160-e9cf-4b95-848d-0336168b038c/volumes"
Oct 30 21:18:31.386089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:31.386001    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-c499bf-5bc85f7679-wd9r9" podUID="bbff0160-e9cf-4b95-848d-0336168b038c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.125:50052\" within 1s: context deadline exceeded"
Oct 30 21:18:31.386242 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:18:31.386122    4918 logging.go:55] [core] [Channel #791 SubChannel #792]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.125:50052", ServerName: "10.132.0.125:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.125:50052: operation was canceled"
Oct 30 21:18:31.388078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:31.388047    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c499bf-86899cd679-qtn5b" podUID="acc8d8d8-6d8c-4c65-adb2-413fa37ee354" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.124:50052\" within 1s: context deadline exceeded"
Oct 30 21:18:31.388227 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:18:31.388126    4918 logging.go:55] [core] [Channel #793 SubChannel #794]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.124:50052", ServerName: "10.132.0.124:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.124:50052: operation was canceled"
Oct 30 21:18:32.524805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.524770    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"]
Oct 30 21:18:32.530069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.528494    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fe0609cc-463d-401e-9117-754584f74c9a" containerName="revoke-keys"
Oct 30 21:18:32.530069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.528518    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0609cc-463d-401e-9117-754584f74c9a" containerName="revoke-keys"
Oct 30 21:18:32.530069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.528597    4918 cpu_manager.go:414] "containerMap: removing container" podUID="bbff0160-e9cf-4b95-848d-0336168b038c" containerName="payload-processing"
Oct 30 21:18:32.530069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.528609    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbff0160-e9cf-4b95-848d-0336168b038c" containerName="payload-processing"
Oct 30 21:18:32.530069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.528695    4918 cpu_manager.go:414] "containerMap: removing container" podUID="acc8d8d8-6d8c-4c65-adb2-413fa37ee354" containerName="payload-pre-processing"
Oct 30 21:18:32.530069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.528704    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc8d8d8-6d8c-4c65-adb2-413fa37ee354" containerName="payload-pre-processing"
Oct 30 21:18:32.530069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.529011    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc8d8d8-6d8c-4c65-adb2-413fa37ee354" containerName="payload-pre-processing"
Oct 30 21:18:32.530069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.529038    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0609cc-463d-401e-9117-754584f74c9a" containerName="revoke-keys"
Oct 30 21:18:32.530069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.529060    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbff0160-e9cf-4b95-848d-0336168b038c" containerName="payload-processing"
Oct 30 21:18:32.535642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.535620    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:32.535877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.535853    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"]
Oct 30 21:18:32.539712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.539686    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:18:32.539848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.539721    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:18:32.539848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.539732    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:18:32.541498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.541479    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:18:32.571918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.571894    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl\" (UID: \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:32.572025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.571957    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w7gx\" (UniqueName: \"kubernetes.io/projected/b9b6b17c-0b3d-4e19-a252-e15dd220681d-kube-api-access-9w7gx\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl\" (UID: \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:32.572066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.572040    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl\" (UID: \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:32.673946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.673916    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl\" (UID: \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:32.674092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.673980    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w7gx\" (UniqueName: \"kubernetes.io/projected/b9b6b17c-0b3d-4e19-a252-e15dd220681d-kube-api-access-9w7gx\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl\" (UID: \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:32.674092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.674019    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl\" (UID: \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:32.674633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.674610    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl\" (UID: \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:32.676515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.676496    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl\" (UID: \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:32.684256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.684232    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w7gx\" (UniqueName: \"kubernetes.io/projected/b9b6b17c-0b3d-4e19-a252-e15dd220681d-kube-api-access-9w7gx\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl\" (UID: \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:32.884824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:32.863459    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:33.213898 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:18:33.213870    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b6b17c_0b3d_4e19_a252_e15dd220681d.slice/crio-9d76d97fe5c74ef791daffa45001fcebc16d7564f6edfc2bd1fd2830b86cf8fc WatchSource:0}: Error finding container 9d76d97fe5c74ef791daffa45001fcebc16d7564f6edfc2bd1fd2830b86cf8fc: Status 404 returned error can't find the container with id 9d76d97fe5c74ef791daffa45001fcebc16d7564f6edfc2bd1fd2830b86cf8fc
Oct 30 21:18:33.214560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:33.214539    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"]
Oct 30 21:18:33.844464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:33.842033    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl" event={"ID":"b9b6b17c-0b3d-4e19-a252-e15dd220681d","Type":"ContainerStarted","Data":"ac77274f2431d7200d45d1d4aa2dbe51e5ab190c68f0f5e5e547195d140977df"}
Oct 30 21:18:33.844464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:33.842108    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl" event={"ID":"b9b6b17c-0b3d-4e19-a252-e15dd220681d","Type":"ContainerStarted","Data":"9d76d97fe5c74ef791daffa45001fcebc16d7564f6edfc2bd1fd2830b86cf8fc"}
Oct 30 21:18:33.865748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:33.865698    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl" podStartSLOduration=1.865682685 podStartE2EDuration="1.865682685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:33.862448802 +0000 UTC m=+1026.909007959" watchObservedRunningTime="2026-10-30 21:18:33.865682685 +0000 UTC m=+1026.912241830"
Oct 30 21:18:34.851993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:34.851958    4918 generic.go:356] "Generic (PLEG): container finished" podID="b9b6b17c-0b3d-4e19-a252-e15dd220681d" containerID="ac77274f2431d7200d45d1d4aa2dbe51e5ab190c68f0f5e5e547195d140977df" exitCode=0
Oct 30 21:18:34.852516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:34.852174    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl" event={"ID":"b9b6b17c-0b3d-4e19-a252-e15dd220681d","Type":"ContainerDied","Data":"ac77274f2431d7200d45d1d4aa2dbe51e5ab190c68f0f5e5e547195d140977df"}
Oct 30 21:18:35.399704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:35.397448    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"]
Oct 30 21:18:35.421417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:35.419569    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c499bf-knb2t"]
Oct 30 21:18:36.034271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.033314    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"]
Oct 30 21:18:36.039848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.039817    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0fc074ef-xd6w9"]
Oct 30 21:18:36.107309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.107286    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:36.280362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.280327    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-openshift-ca\") pod \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\" (UID: \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\") "
Oct 30 21:18:36.280570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.280492    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-xks-ca\") pod \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\" (UID: \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\") "
Oct 30 21:18:36.280655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.280618    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b6b17c-0b3d-4e19-a252-e15dd220681d-kube-api-access-9w7gx\" (UniqueName: \"kubernetes.io/projected/b9b6b17c-0b3d-4e19-a252-e15dd220681d-kube-api-access-9w7gx\") pod \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\" (UID: \"b9b6b17c-0b3d-4e19-a252-e15dd220681d\") "
Oct 30 21:18:36.280740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.280713    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-openshift-ca" pod "b9b6b17c-0b3d-4e19-a252-e15dd220681d" (UID: "b9b6b17c-0b3d-4e19-a252-e15dd220681d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:36.281161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.281140    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:36.282781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.282759    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-xks-ca" pod "b9b6b17c-0b3d-4e19-a252-e15dd220681d" (UID: "b9b6b17c-0b3d-4e19-a252-e15dd220681d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:36.282883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.282816    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b6b17c-0b3d-4e19-a252-e15dd220681d-kube-api-access-9w7gx" pod "b9b6b17c-0b3d-4e19-a252-e15dd220681d" (UID: "b9b6b17c-0b3d-4e19-a252-e15dd220681d"). InnerVolumeSpecName "kube-api-access-9w7gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:36.382286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.382255    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b6b17c-0b3d-4e19-a252-e15dd220681d-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:36.382286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.382283    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w7gx\" (UniqueName: \"kubernetes.io/projected/b9b6b17c-0b3d-4e19-a252-e15dd220681d-kube-api-access-9w7gx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:36.782708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.782677    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"]
Oct 30 21:18:36.782941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.782904    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy" containerID="cri-o://79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5" gracePeriod=30
Oct 30 21:18:36.985125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.984911    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl" event={"ID":"b9b6b17c-0b3d-4e19-a252-e15dd220681d","Type":"ContainerDied","Data":"9d76d97fe5c74ef791daffa45001fcebc16d7564f6edfc2bd1fd2830b86cf8fc"}
Oct 30 21:18:36.985749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.985727    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d76d97fe5c74ef791daffa45001fcebc16d7564f6edfc2bd1fd2830b86cf8fc"
Oct 30 21:18:36.985962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:36.985949    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"
Oct 30 21:18:37.165251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.164723    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13997d71-c50a-4b3b-bf63-e52399720783" path="/var/lib/kubelet/pods/13997d71-c50a-4b3b-bf63-e52399720783/volumes"
Oct 30 21:18:37.165251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.165217    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7db77f-68bb-4b4a-8cbe-66beee304169" path="/var/lib/kubelet/pods/ae7db77f-68bb-4b4a-8cbe-66beee304169/volumes"
Oct 30 21:18:37.437494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.434800    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"]
Oct 30 21:18:37.437494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.436029    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b9b6b17c-0b3d-4e19-a252-e15dd220681d" containerName="revoke-keys"
Oct 30 21:18:37.437494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.436049    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b6b17c-0b3d-4e19-a252-e15dd220681d" containerName="revoke-keys"
Oct 30 21:18:37.437494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.436218    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b6b17c-0b3d-4e19-a252-e15dd220681d" containerName="revoke-keys"
Oct 30 21:18:37.453413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.447832    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"]
Oct 30 21:18:37.453413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.447936    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:37.453413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.452674    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:18:37.453725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.453489    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:18:37.457748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.455159    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfmgt\" (UniqueName: \"kubernetes.io/projected/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-kube-api-access-mfmgt\") pod \"maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh\" (UID: \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:37.457748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.455276    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh\" (UID: \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:37.457748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.455321    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh\" (UID: \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:37.488451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.484862    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:18:37.488451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.484937    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:18:37.562445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.562178    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfmgt\" (UniqueName: \"kubernetes.io/projected/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-kube-api-access-mfmgt\") pod \"maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh\" (UID: \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:37.562445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.562354    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh\" (UID: \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:37.562739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.562480    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh\" (UID: \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:37.569032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.568999    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh\" (UID: \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:37.574481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.574451    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh\" (UID: \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:37.578151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.578122    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfmgt\" (UniqueName: \"kubernetes.io/projected/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-kube-api-access-mfmgt\") pod \"maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh\" (UID: \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:37.790513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:37.790482    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:38.072206 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:18:38.072180    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9c97df_e472_41ac_b9a8_6cda15da8dd4.slice/crio-88d99f0f7ee9348a69f159a98c2f5e0f1616006d8b965631eec523f371b43027 WatchSource:0}: Error finding container 88d99f0f7ee9348a69f159a98c2f5e0f1616006d8b965631eec523f371b43027: Status 404 returned error can't find the container with id 88d99f0f7ee9348a69f159a98c2f5e0f1616006d8b965631eec523f371b43027
Oct 30 21:18:38.073329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:38.073308    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"]
Oct 30 21:18:39.033326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:39.033275    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh" event={"ID":"6e9c97df-e472-41ac-b9a8-6cda15da8dd4","Type":"ContainerStarted","Data":"f091cb6ff01501804755445088bdbc7bd65a62323a75818399e92384b7d2cc5b"}
Oct 30 21:18:39.033881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:39.033406    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh" event={"ID":"6e9c97df-e472-41ac-b9a8-6cda15da8dd4","Type":"ContainerStarted","Data":"88d99f0f7ee9348a69f159a98c2f5e0f1616006d8b965631eec523f371b43027"}
Oct 30 21:18:39.058412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:39.057779    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh" podStartSLOduration=2.057758375 podStartE2EDuration="2.057758375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:39.052973273 +0000 UTC m=+1032.099532443" watchObservedRunningTime="2026-10-30 21:18:39.057758375 +0000 UTC m=+1032.104317516"
Oct 30 21:18:40.048415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:40.047807    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"]
Oct 30 21:18:40.057846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:40.057800    4918 generic.go:356] "Generic (PLEG): container finished" podID="6e9c97df-e472-41ac-b9a8-6cda15da8dd4" containerID="f091cb6ff01501804755445088bdbc7bd65a62323a75818399e92384b7d2cc5b" exitCode=0
Oct 30 21:18:40.058408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:40.058033    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh" event={"ID":"6e9c97df-e472-41ac-b9a8-6cda15da8dd4","Type":"ContainerDied","Data":"f091cb6ff01501804755445088bdbc7bd65a62323a75818399e92384b7d2cc5b"}
Oct 30 21:18:40.061085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:40.060991    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b26e13e-pjrwc"]
Oct 30 21:18:41.079306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.079279    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19565d3c-4c31-4118-bcc9-8420563f488f" path="/var/lib/kubelet/pods/19565d3c-4c31-4118-bcc9-8420563f488f/volumes"
Oct 30 21:18:41.515929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.515529    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-smshz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:18:41.515929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.515627    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" podUID="a0439b0d-ff5f-49ab-87c3-afada363adcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:18:41.619416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.603882    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:41.777643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.777532    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-kube-api-access-mfmgt\" (UniqueName: \"kubernetes.io/projected/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-kube-api-access-mfmgt\") pod \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\" (UID: \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\") "
Oct 30 21:18:41.780242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.777890    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-xks-ca\") pod \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\" (UID: \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\") "
Oct 30 21:18:41.780864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.780832    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-openshift-ca\") pod \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\" (UID: \"6e9c97df-e472-41ac-b9a8-6cda15da8dd4\") "
Oct 30 21:18:41.785113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.785091    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-openshift-ca" pod "6e9c97df-e472-41ac-b9a8-6cda15da8dd4" (UID: "6e9c97df-e472-41ac-b9a8-6cda15da8dd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:41.790850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.787353    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-xks-ca" pod "6e9c97df-e472-41ac-b9a8-6cda15da8dd4" (UID: "6e9c97df-e472-41ac-b9a8-6cda15da8dd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:41.799253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.799216    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-kube-api-access-mfmgt" pod "6e9c97df-e472-41ac-b9a8-6cda15da8dd4" (UID: "6e9c97df-e472-41ac-b9a8-6cda15da8dd4"). InnerVolumeSpecName "kube-api-access-mfmgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:41.887418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.884988    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:41.887418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.885025    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfmgt\" (UniqueName: \"kubernetes.io/projected/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-kube-api-access-mfmgt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:41.887418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:41.885040    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9c97df-e472-41ac-b9a8-6cda15da8dd4-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.076797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.076769    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:18:42.081510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.081489    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"
Oct 30 21:18:42.081882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.081536    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh" event={"ID":"6e9c97df-e472-41ac-b9a8-6cda15da8dd4","Type":"ContainerDied","Data":"88d99f0f7ee9348a69f159a98c2f5e0f1616006d8b965631eec523f371b43027"}
Oct 30 21:18:42.081882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.081578    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88d99f0f7ee9348a69f159a98c2f5e0f1616006d8b965631eec523f371b43027"
Oct 30 21:18:42.083562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.083531    4918 generic.go:356] "Generic (PLEG): container finished" podID="735fc195-18ba-475c-b71a-65711bdc589a" containerID="79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5" exitCode=0
Oct 30 21:18:42.083684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.083567    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"
Oct 30 21:18:42.083684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.083573    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" event={"ID":"735fc195-18ba-475c-b71a-65711bdc589a","Type":"ContainerDied","Data":"79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5"}
Oct 30 21:18:42.083684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.083656    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6" event={"ID":"735fc195-18ba-475c-b71a-65711bdc589a","Type":"ContainerDied","Data":"30b79ca700a2f990e935ab94b43354417e7c5b612d7933486322c4e85bce45f2"}
Oct 30 21:18:42.083793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.083706    4918 scope.go:122] "RemoveContainer" containerID="79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5"
Oct 30 21:18:42.108507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.108487    4918 scope.go:122] "RemoveContainer" containerID="35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f"
Oct 30 21:18:42.124856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.124831    4918 scope.go:122] "RemoveContainer" containerID="79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5"
Oct 30 21:18:42.125262 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:18:42.125231    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5\": container with ID starting with 79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5 not found: ID does not exist" containerID="79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5"
Oct 30 21:18:42.125414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.125265    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5"} err="failed to get container status \"79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5\": rpc error: code = NotFound desc = could not find container \"79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5\": container with ID starting with 79da82a4d1450dcdf9c93d707c97748725fd9cd1945de22c9ccef55fd3145db5 not found: ID does not exist"
Oct 30 21:18:42.125414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.125284    4918 scope.go:122] "RemoveContainer" containerID="35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f"
Oct 30 21:18:42.125655 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:18:42.125627    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f\": container with ID starting with 35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f not found: ID does not exist" containerID="35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f"
Oct 30 21:18:42.125723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.125662    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f"} err="failed to get container status \"35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f\": rpc error: code = NotFound desc = could not find container \"35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f\": container with ID starting with 35bda3d4899c12f5d6ad8a5e2c3716ae64d30dd381825fa6718af1e478db2f1f not found: ID does not exist"
Oct 30 21:18:42.191477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.191443    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/735fc195-18ba-475c-b71a-65711bdc589a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735fc195-18ba-475c-b71a-65711bdc589a-istio-podinfo\") pod \"735fc195-18ba-475c-b71a-65711bdc589a\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") "
Oct 30 21:18:42.191682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.191579    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-certs\") pod \"735fc195-18ba-475c-b71a-65711bdc589a\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") "
Oct 30 21:18:42.191682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.191668    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-istio-token\" (UniqueName: \"kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-istio-token\") pod \"735fc195-18ba-475c-b71a-65711bdc589a\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") "
Oct 30 21:18:42.191803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.191713    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-socket\") pod \"735fc195-18ba-475c-b71a-65711bdc589a\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") "
Oct 30 21:18:42.191803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.191752    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-kube-api-access-w8tds\" (UniqueName: \"kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-kube-api-access-w8tds\") pod \"735fc195-18ba-475c-b71a-65711bdc589a\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") "
Oct 30 21:18:42.191913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.191848    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-certs" pod "735fc195-18ba-475c-b71a-65711bdc589a" (UID: "735fc195-18ba-475c-b71a-65711bdc589a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:42.191913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.191851    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-data\") pod \"735fc195-18ba-475c-b71a-65711bdc589a\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") "
Oct 30 21:18:42.192015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.191943    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/735fc195-18ba-475c-b71a-65711bdc589a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735fc195-18ba-475c-b71a-65711bdc589a-istiod-ca-cert\") pod \"735fc195-18ba-475c-b71a-65711bdc589a\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") "
Oct 30 21:18:42.192071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.192018    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-data" pod "735fc195-18ba-475c-b71a-65711bdc589a" (UID: "735fc195-18ba-475c-b71a-65711bdc589a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:42.192071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.192054    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-socket" pod "735fc195-18ba-475c-b71a-65711bdc589a" (UID: "735fc195-18ba-475c-b71a-65711bdc589a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:42.192173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.192078    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-envoy\") pod \"735fc195-18ba-475c-b71a-65711bdc589a\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") "
Oct 30 21:18:42.192173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.192115    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-credential-socket\") pod \"735fc195-18ba-475c-b71a-65711bdc589a\" (UID: \"735fc195-18ba-475c-b71a-65711bdc589a\") "
Oct 30 21:18:42.192342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.192322    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735fc195-18ba-475c-b71a-65711bdc589a-istiod-ca-cert" pod "735fc195-18ba-475c-b71a-65711bdc589a" (UID: "735fc195-18ba-475c-b71a-65711bdc589a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:42.192497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.192473    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-credential-socket" pod "735fc195-18ba-475c-b71a-65711bdc589a" (UID: "735fc195-18ba-475c-b71a-65711bdc589a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:42.192820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.192794    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735fc195-18ba-475c-b71a-65711bdc589a-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.192880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.192818    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.192880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.192833    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.192880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.192847    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.192880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.192860    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.194938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.194909    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-envoy" pod "735fc195-18ba-475c-b71a-65711bdc589a" (UID: "735fc195-18ba-475c-b71a-65711bdc589a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:18:42.195570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.195538    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/735fc195-18ba-475c-b71a-65711bdc589a-istio-podinfo" pod "735fc195-18ba-475c-b71a-65711bdc589a" (UID: "735fc195-18ba-475c-b71a-65711bdc589a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:18:42.195662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.195593    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-istio-token" pod "735fc195-18ba-475c-b71a-65711bdc589a" (UID: "735fc195-18ba-475c-b71a-65711bdc589a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:42.196155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.196129    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-kube-api-access-w8tds" pod "735fc195-18ba-475c-b71a-65711bdc589a" (UID: "735fc195-18ba-475c-b71a-65711bdc589a"). InnerVolumeSpecName "kube-api-access-w8tds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:42.294318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.294281    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.294318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.294321    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8tds\" (UniqueName: \"kubernetes.io/projected/735fc195-18ba-475c-b71a-65711bdc589a-kube-api-access-w8tds\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.294595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.294338    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735fc195-18ba-475c-b71a-65711bdc589a-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.294595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.294351    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735fc195-18ba-475c-b71a-65711bdc589a-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:42.386502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.386425    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"]
Oct 30 21:18:42.387501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.387481    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6e9c97df-e472-41ac-b9a8-6cda15da8dd4" containerName="revoke-keys"
Oct 30 21:18:42.387556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.387505    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9c97df-e472-41ac-b9a8-6cda15da8dd4" containerName="revoke-keys"
Oct 30 21:18:42.387556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.387531    4918 cpu_manager.go:414] "containerMap: removing container" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy"
Oct 30 21:18:42.387556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.387541    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy"
Oct 30 21:18:42.387656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.387561    4918 cpu_manager.go:414] "containerMap: removing container" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy"
Oct 30 21:18:42.387656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.387570    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy"
Oct 30 21:18:42.387756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.387745    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9c97df-e472-41ac-b9a8-6cda15da8dd4" containerName="revoke-keys"
Oct 30 21:18:42.387791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.387764    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy"
Oct 30 21:18:42.387791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.387773    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="735fc195-18ba-475c-b71a-65711bdc589a" containerName="istio-proxy"
Oct 30 21:18:42.392070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.392047    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"
Oct 30 21:18:42.396170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.396143    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a82ca5-2768-4089-a95e-453351c37254-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l\" (UID: \"20a82ca5-2768-4089-a95e-453351c37254\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"
Oct 30 21:18:42.396538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.396511    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7wvr\" (UniqueName: \"kubernetes.io/projected/20a82ca5-2768-4089-a95e-453351c37254-kube-api-access-p7wvr\") pod \"payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l\" (UID: \"20a82ca5-2768-4089-a95e-453351c37254\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"
Oct 30 21:18:42.396953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.396932    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0c4aac-dockercfg-bvx8n\""
Oct 30 21:18:42.397061 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.396977    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0c4aac\""
Oct 30 21:18:42.402105 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.401948    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"]
Oct 30 21:18:42.431849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.431817    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"]
Oct 30 21:18:42.437050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.437027    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-86671d-openshift-default-594fd777d9-c6xs6"]
Oct 30 21:18:42.516548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.512311    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7wvr\" (UniqueName: \"kubernetes.io/projected/20a82ca5-2768-4089-a95e-453351c37254-kube-api-access-p7wvr\") pod \"payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l\" (UID: \"20a82ca5-2768-4089-a95e-453351c37254\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"
Oct 30 21:18:42.516548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.512558    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a82ca5-2768-4089-a95e-453351c37254-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l\" (UID: \"20a82ca5-2768-4089-a95e-453351c37254\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"
Oct 30 21:18:42.516548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.513633    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a82ca5-2768-4089-a95e-453351c37254-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l\" (UID: \"20a82ca5-2768-4089-a95e-453351c37254\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"
Oct 30 21:18:42.566129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.566097    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7wvr\" (UniqueName: \"kubernetes.io/projected/20a82ca5-2768-4089-a95e-453351c37254-kube-api-access-p7wvr\") pod \"payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l\" (UID: \"20a82ca5-2768-4089-a95e-453351c37254\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"
Oct 30 21:18:42.712329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:42.712250    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"
Oct 30 21:18:43.125767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:43.124994    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="735fc195-18ba-475c-b71a-65711bdc589a" path="/var/lib/kubelet/pods/735fc195-18ba-475c-b71a-65711bdc589a/volumes"
Oct 30 21:18:43.156973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:43.156910    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"]
Oct 30 21:18:43.173791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:43.173759    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"]
Oct 30 21:18:43.189132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:43.189110    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"]
Oct 30 21:18:44.173283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:44.173183    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l" event={"ID":"20a82ca5-2768-4089-a95e-453351c37254","Type":"ContainerStarted","Data":"05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7"}
Oct 30 21:18:44.173681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:44.173326    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"
Oct 30 21:18:44.173681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:44.173353    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l" event={"ID":"20a82ca5-2768-4089-a95e-453351c37254","Type":"ContainerStarted","Data":"8306383620bc124e6efa9c6b252eb3157aa56e0bb9e4c8f3a1ca854bcc45be68"}
Oct 30 21:18:44.198853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:44.198800    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l" podStartSLOduration=2.198782551 podStartE2EDuration="2.198782551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:44.194550719 +0000 UTC m=+1037.241109891" watchObservedRunningTime="2026-10-30 21:18:44.198782551 +0000 UTC m=+1037.245341697"
Oct 30 21:18:47.565044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:47.565009    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"]
Oct 30 21:18:47.578669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:47.578641    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"]
Oct 30 21:18:48.176421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.175466    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb" podUID="1495e8dd-272b-4be0-882e-5d89f2b5223a" containerName="payload-processing" containerID="cri-o://11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5" gracePeriod=25
Oct 30 21:18:48.195684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.195500    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf" podUID="ad901042-4981-479b-98a3-8c6b08db50a2" containerName="payload-pre-processing" containerID="cri-o://20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca" gracePeriod=25
Oct 30 21:18:48.225915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.225887    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"]
Oct 30 21:18:48.230685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.230660    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:48.239747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.239719    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:18:48.239939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.239874    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:18:48.240009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.239975    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:18:48.240137 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.240072    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:18:48.253426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.248367    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"]
Oct 30 21:18:48.356423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.326978    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nnsw\" (UniqueName: \"kubernetes.io/projected/8c40f532-0846-4db6-8af6-fef82a28c5ac-kube-api-access-6nnsw\") pod \"maas-api-revoke-sub-58933cb1-g4skp\" (UID: \"8c40f532-0846-4db6-8af6-fef82a28c5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:48.356423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.327036    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58933cb1-g4skp\" (UID: \"8c40f532-0846-4db6-8af6-fef82a28c5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:48.356423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.327154    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58933cb1-g4skp\" (UID: \"8c40f532-0846-4db6-8af6-fef82a28c5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:48.429084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.429005    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nnsw\" (UniqueName: \"kubernetes.io/projected/8c40f532-0846-4db6-8af6-fef82a28c5ac-kube-api-access-6nnsw\") pod \"maas-api-revoke-sub-58933cb1-g4skp\" (UID: \"8c40f532-0846-4db6-8af6-fef82a28c5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:48.429084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.429064    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58933cb1-g4skp\" (UID: \"8c40f532-0846-4db6-8af6-fef82a28c5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:48.429266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.429180    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58933cb1-g4skp\" (UID: \"8c40f532-0846-4db6-8af6-fef82a28c5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:48.430659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.430632    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58933cb1-g4skp\" (UID: \"8c40f532-0846-4db6-8af6-fef82a28c5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:48.432255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.432227    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58933cb1-g4skp\" (UID: \"8c40f532-0846-4db6-8af6-fef82a28c5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:48.441819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.441775    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nnsw\" (UniqueName: \"kubernetes.io/projected/8c40f532-0846-4db6-8af6-fef82a28c5ac-kube-api-access-6nnsw\") pod \"maas-api-revoke-sub-58933cb1-g4skp\" (UID: \"8c40f532-0846-4db6-8af6-fef82a28c5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:48.554353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.554319    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:48.947566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:48.947535    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"]
Oct 30 21:18:48.948925 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:18:48.948897    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c40f532_0846_4db6_8af6_fef82a28c5ac.slice/crio-0933e4c24cfe7f67c57eb80a5a9e22655e33d546bc2aba8c2070f06736654d11 WatchSource:0}: Error finding container 0933e4c24cfe7f67c57eb80a5a9e22655e33d546bc2aba8c2070f06736654d11: Status 404 returned error can't find the container with id 0933e4c24cfe7f67c57eb80a5a9e22655e33d546bc2aba8c2070f06736654d11
Oct 30 21:18:49.221820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:49.221792    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp" event={"ID":"8c40f532-0846-4db6-8af6-fef82a28c5ac","Type":"ContainerStarted","Data":"4440e7edd25f05580a9912e8cfb892d1f6c0fb9ab669118b9847f739992363ce"}
Oct 30 21:18:49.221937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:49.221859    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp" event={"ID":"8c40f532-0846-4db6-8af6-fef82a28c5ac","Type":"ContainerStarted","Data":"0933e4c24cfe7f67c57eb80a5a9e22655e33d546bc2aba8c2070f06736654d11"}
Oct 30 21:18:49.244246 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:49.244198    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp" podStartSLOduration=1.244185112 podStartE2EDuration="1.244185112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:49.239321177 +0000 UTC m=+1042.285880322" watchObservedRunningTime="2026-10-30 21:18:49.244185112 +0000 UTC m=+1042.290744257"
Oct 30 21:18:50.025939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:50.025905    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"]
Oct 30 21:18:50.030654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:50.030624    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83f16a28-t9mbd"]
Oct 30 21:18:50.233012 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:50.232976    4918 generic.go:356] "Generic (PLEG): container finished" podID="8c40f532-0846-4db6-8af6-fef82a28c5ac" containerID="4440e7edd25f05580a9912e8cfb892d1f6c0fb9ab669118b9847f739992363ce" exitCode=0
Oct 30 21:18:50.233200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:50.233039    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp" event={"ID":"8c40f532-0846-4db6-8af6-fef82a28c5ac","Type":"ContainerDied","Data":"4440e7edd25f05580a9912e8cfb892d1f6c0fb9ab669118b9847f739992363ce"}
Oct 30 21:18:51.078054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:51.078020    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e201359f-1235-4bf4-a085-b88dc962e9b2" path="/var/lib/kubelet/pods/e201359f-1235-4bf4-a085-b88dc962e9b2/volumes"
Oct 30 21:18:51.508141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:51.508120    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:51.586362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:51.586328    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-openshift-ca\") pod \"8c40f532-0846-4db6-8af6-fef82a28c5ac\" (UID: \"8c40f532-0846-4db6-8af6-fef82a28c5ac\") "
Oct 30 21:18:51.586575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:51.586556    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-xks-ca\") pod \"8c40f532-0846-4db6-8af6-fef82a28c5ac\" (UID: \"8c40f532-0846-4db6-8af6-fef82a28c5ac\") "
Oct 30 21:18:51.586623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:51.586597    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c40f532-0846-4db6-8af6-fef82a28c5ac-kube-api-access-6nnsw\" (UniqueName: \"kubernetes.io/projected/8c40f532-0846-4db6-8af6-fef82a28c5ac-kube-api-access-6nnsw\") pod \"8c40f532-0846-4db6-8af6-fef82a28c5ac\" (UID: \"8c40f532-0846-4db6-8af6-fef82a28c5ac\") "
Oct 30 21:18:51.586745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:51.586727    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-openshift-ca" pod "8c40f532-0846-4db6-8af6-fef82a28c5ac" (UID: "8c40f532-0846-4db6-8af6-fef82a28c5ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:51.587169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:51.587147    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:51.588779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:51.588752    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-xks-ca" pod "8c40f532-0846-4db6-8af6-fef82a28c5ac" (UID: "8c40f532-0846-4db6-8af6-fef82a28c5ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:18:51.588871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:51.588823    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c40f532-0846-4db6-8af6-fef82a28c5ac-kube-api-access-6nnsw" pod "8c40f532-0846-4db6-8af6-fef82a28c5ac" (UID: "8c40f532-0846-4db6-8af6-fef82a28c5ac"). InnerVolumeSpecName "kube-api-access-6nnsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:51.688921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:51.688892    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c40f532-0846-4db6-8af6-fef82a28c5ac-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:51.688921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:51.688922    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nnsw\" (UniqueName: \"kubernetes.io/projected/8c40f532-0846-4db6-8af6-fef82a28c5ac-kube-api-access-6nnsw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:52.250733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:52.250705    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"
Oct 30 21:18:52.250733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:52.250724    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp" event={"ID":"8c40f532-0846-4db6-8af6-fef82a28c5ac","Type":"ContainerDied","Data":"0933e4c24cfe7f67c57eb80a5a9e22655e33d546bc2aba8c2070f06736654d11"}
Oct 30 21:18:52.251163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:52.250764    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0933e4c24cfe7f67c57eb80a5a9e22655e33d546bc2aba8c2070f06736654d11"
Oct 30 21:18:52.566444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:52.566330    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h" podUID="ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4" containerName="payload-processing" containerID="cri-o://fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a" gracePeriod=25
Oct 30 21:18:52.579558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:52.579525    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp" podUID="13e099b0-8c43-4d95-b056-fed2463dcd53" containerName="payload-pre-processing" containerID="cri-o://232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8" gracePeriod=25
Oct 30 21:18:53.005881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.005852    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"
Oct 30 21:18:53.014065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.014042    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"
Oct 30 21:18:53.124574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.124486    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e099b0-8c43-4d95-b056-fed2463dcd53-kube-api-access-x7fr6\" (UniqueName: \"kubernetes.io/projected/13e099b0-8c43-4d95-b056-fed2463dcd53-kube-api-access-x7fr6\") pod \"13e099b0-8c43-4d95-b056-fed2463dcd53\" (UID: \"13e099b0-8c43-4d95-b056-fed2463dcd53\") "
Oct 30 21:18:53.124735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.124594    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-kube-api-access-7g6mh\" (UniqueName: \"kubernetes.io/projected/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-kube-api-access-7g6mh\") pod \"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4\" (UID: \"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4\") "
Oct 30 21:18:53.124735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.124706    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e099b0-8c43-4d95-b056-fed2463dcd53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e099b0-8c43-4d95-b056-fed2463dcd53-plugins-config-volume\") pod \"13e099b0-8c43-4d95-b056-fed2463dcd53\" (UID: \"13e099b0-8c43-4d95-b056-fed2463dcd53\") "
Oct 30 21:18:53.124874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.124741    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-plugins-config-volume\") pod \"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4\" (UID: \"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4\") "
Oct 30 21:18:53.125134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.125109    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e099b0-8c43-4d95-b056-fed2463dcd53-plugins-config-volume" pod "13e099b0-8c43-4d95-b056-fed2463dcd53" (UID: "13e099b0-8c43-4d95-b056-fed2463dcd53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:53.125211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.125157    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-plugins-config-volume" pod "ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4" (UID: "ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:18:53.125642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.125620    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e099b0-8c43-4d95-b056-fed2463dcd53-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:53.125695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.125680    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:53.126919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.126895    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-kube-api-access-7g6mh" pod "ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4" (UID: "ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4"). InnerVolumeSpecName "kube-api-access-7g6mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:53.127020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.126973    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e099b0-8c43-4d95-b056-fed2463dcd53-kube-api-access-x7fr6" pod "13e099b0-8c43-4d95-b056-fed2463dcd53" (UID: "13e099b0-8c43-4d95-b056-fed2463dcd53"). InnerVolumeSpecName "kube-api-access-x7fr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:18:53.227462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.227433    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7fr6\" (UniqueName: \"kubernetes.io/projected/13e099b0-8c43-4d95-b056-fed2463dcd53-kube-api-access-x7fr6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:53.227462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.227463    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g6mh\" (UniqueName: \"kubernetes.io/projected/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4-kube-api-access-7g6mh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:18:53.261775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.261746    4918 generic.go:356] "Generic (PLEG): container finished" podID="ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4" containerID="fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a" exitCode=0
Oct 30 21:18:53.262251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.261793    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"
Oct 30 21:18:53.262251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.261845    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h" event={"ID":"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4","Type":"ContainerDied","Data":"fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a"}
Oct 30 21:18:53.262251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.261911    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h" event={"ID":"ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4","Type":"ContainerDied","Data":"9d15f1447fdda49da2b7bc9f4955cd95db30515e71bf9f2fde573af899454cca"}
Oct 30 21:18:53.262251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.261943    4918 scope.go:122] "RemoveContainer" containerID="fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a"
Oct 30 21:18:53.263700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.263680    4918 generic.go:356] "Generic (PLEG): container finished" podID="13e099b0-8c43-4d95-b056-fed2463dcd53" containerID="232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8" exitCode=0
Oct 30 21:18:53.263845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.263719    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp" event={"ID":"13e099b0-8c43-4d95-b056-fed2463dcd53","Type":"ContainerDied","Data":"232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8"}
Oct 30 21:18:53.263845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.263732    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"
Oct 30 21:18:53.263845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.263759    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp" event={"ID":"13e099b0-8c43-4d95-b056-fed2463dcd53","Type":"ContainerDied","Data":"b6957a7893c617a50bf2c82054ef5fe9ba8472927d5f6b1d9a0e221e105c4f78"}
Oct 30 21:18:53.274344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.274328    4918 scope.go:122] "RemoveContainer" containerID="fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a"
Oct 30 21:18:53.274635 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:18:53.274616    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a\": container with ID starting with fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a not found: ID does not exist" containerID="fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a"
Oct 30 21:18:53.274687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.274645    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a"} err="failed to get container status \"fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a\": rpc error: code = NotFound desc = could not find container \"fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a\": container with ID starting with fe2b006307f883838b3e04a3351be5ea747e1938b111338369b5b83a7f06c59a not found: ID does not exist"
Oct 30 21:18:53.274687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.274660    4918 scope.go:122] "RemoveContainer" containerID="232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8"
Oct 30 21:18:53.284788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.284772    4918 scope.go:122] "RemoveContainer" containerID="232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8"
Oct 30 21:18:53.285049 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:18:53.285031    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8\": container with ID starting with 232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8 not found: ID does not exist" containerID="232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8"
Oct 30 21:18:53.285104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.285057    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8"} err="failed to get container status \"232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8\": rpc error: code = NotFound desc = could not find container \"232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8\": container with ID starting with 232dfff6f02f0464df25417ee2ef01f2e6cf70b76fd538bd02a1bcc2c2beb7e8 not found: ID does not exist"
Oct 30 21:18:53.291091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.291068    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"]
Oct 30 21:18:53.295492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.295472    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h"]
Oct 30 21:18:53.304808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.304785    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"]
Oct 30 21:18:53.309764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.309732    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp"]
Oct 30 21:18:53.861852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.861755    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b3d1de-867fb6dfc9-pvkkp" podUID="13e099b0-8c43-4d95-b056-fed2463dcd53" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.127:50052\" within 1s: context deadline exceeded"
Oct 30 21:18:53.861996 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:18:53.861894    4918 logging.go:55] [core] [Channel #849 SubChannel #850]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.127:50052", ServerName: "10.132.0.127:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.127:50052: operation was canceled"
Oct 30 21:18:53.864902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:53.864873    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-b3d1de-65f88b6f6c-qfc4h" podUID="ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.128:50052\" within 1s: context deadline exceeded"
Oct 30 21:18:53.865017 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:18:53.864952    4918 logging.go:55] [core] [Channel #851 SubChannel #852]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.128:50052", ServerName: "10.132.0.128:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.128:50052: operation was canceled"
Oct 30 21:18:54.978943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:54.978906    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb" podUID="1495e8dd-272b-4be0-882e-5d89f2b5223a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:18:54.979323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:54.978950    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf" podUID="ad901042-4981-479b-98a3-8c6b08db50a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:18:55.078084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:55.078049    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e099b0-8c43-4d95-b056-fed2463dcd53" path="/var/lib/kubelet/pods/13e099b0-8c43-4d95-b056-fed2463dcd53/volumes"
Oct 30 21:18:55.078618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:55.078601    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4" path="/var/lib/kubelet/pods/ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4/volumes"
Oct 30 21:18:55.186256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:18:55.186227    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"
Oct 30 21:18:59.198996 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:18:59.198959    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:19:02.079346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.079312    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"]
Oct 30 21:19:02.080900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.080856    4918 cpu_manager.go:414] "containerMap: removing container" podUID="13e099b0-8c43-4d95-b056-fed2463dcd53" containerName="payload-pre-processing"
Oct 30 21:19:02.081074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.081059    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e099b0-8c43-4d95-b056-fed2463dcd53" containerName="payload-pre-processing"
Oct 30 21:19:02.081178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.081168    4918 cpu_manager.go:414] "containerMap: removing container" podUID="8c40f532-0846-4db6-8af6-fef82a28c5ac" containerName="revoke-keys"
Oct 30 21:19:02.081264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.081254    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c40f532-0846-4db6-8af6-fef82a28c5ac" containerName="revoke-keys"
Oct 30 21:19:02.081430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.081417    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4" containerName="payload-processing"
Oct 30 21:19:02.081537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.081525    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4" containerName="payload-processing"
Oct 30 21:19:02.081780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.081767    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef466c2f-9fc8-4ac3-9ded-f89715a5cfa4" containerName="payload-processing"
Oct 30 21:19:02.081892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.081881    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e099b0-8c43-4d95-b056-fed2463dcd53" containerName="payload-pre-processing"
Oct 30 21:19:02.082026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.081985    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c40f532-0846-4db6-8af6-fef82a28c5ac" containerName="revoke-keys"
Oct 30 21:19:02.088671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.088641    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.099447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.099317    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e494a9-openshift-default-dockercfg-kh8q6\""
Oct 30 21:19:02.114015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.112563    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"]
Oct 30 21:19:02.163215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.163182    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6qf9\" (UniqueName: \"kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-kube-api-access-w6qf9\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.163376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.163236    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-credential-socket\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.163376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.163313    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-envoy\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.163376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.163347    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-token\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.163376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.163371    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istiod-ca-cert\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.163560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.163530    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-socket\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.163616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.163583    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-certs\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.163678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.163638    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-data\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.163737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.163721    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-podinfo\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.280618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.280584    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-certs\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.280938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.280901    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-data\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.281101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.281085    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-podinfo\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.281632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.281609    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qf9\" (UniqueName: \"kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-kube-api-access-w6qf9\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.287441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.281817    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-credential-socket\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.287707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.287676    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-envoy\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.287863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.287845    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-token\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.288100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.288000    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istiod-ca-cert\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.288371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.288352    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-socket\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.307414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.289048    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-socket\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.308334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.293600    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-certs\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.308537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.293805    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-data\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.310193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.310160    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-credential-socket\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.311186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.311152    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istiod-ca-cert\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.328713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.328648    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-podinfo\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.333516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.329777    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6qf9\" (UniqueName: \"kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-kube-api-access-w6qf9\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.338455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.335209    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-envoy\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.357261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.357193    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-token\") pod \"e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") " pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.418697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.418666    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:02.743207 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:19:02.743178    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c28dba_cda4_44cd_8f7e_ac1774dfe309.slice/crio-a8a97a346ea9c8e198ed90be3163c4481d66fbc3c6bdbdd19ea119a2250c80f8 WatchSource:0}: Error finding container a8a97a346ea9c8e198ed90be3163c4481d66fbc3c6bdbdd19ea119a2250c80f8: Status 404 returned error can't find the container with id a8a97a346ea9c8e198ed90be3163c4481d66fbc3c6bdbdd19ea119a2250c80f8
Oct 30 21:19:02.744077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.744056    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"]
Oct 30 21:19:02.746357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.746326    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:19:02.746480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.746422    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:19:02.746480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:02.746455    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:19:03.525757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:03.525718    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h" event={"ID":"19c28dba-cda4-44cd-8f7e-ac1774dfe309","Type":"ContainerStarted","Data":"7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19"}
Oct 30 21:19:03.545729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:03.545689    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h" event={"ID":"19c28dba-cda4-44cd-8f7e-ac1774dfe309","Type":"ContainerStarted","Data":"a8a97a346ea9c8e198ed90be3163c4481d66fbc3c6bdbdd19ea119a2250c80f8"}
Oct 30 21:19:03.618420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:03.597966    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h" podStartSLOduration=1.5979458389999999 podStartE2EDuration="1.597945839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:03.569972419 +0000 UTC m=+1056.616531565" watchObservedRunningTime="2026-10-30 21:19:03.597945839 +0000 UTC m=+1056.644504985"
Oct 30 21:19:04.110528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:04.110448    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"]
Oct 30 21:19:04.128454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:04.118863    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b3d1de-km6fh"]
Oct 30 21:19:04.428791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:04.428711    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:04.459434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:04.450882    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:04.567437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:04.565633    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:04.567437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:04.566914    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:19:04.980546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:04.980502    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf" podUID="ad901042-4981-479b-98a3-8c6b08db50a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:19:04.980546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:04.980520    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb" podUID="1495e8dd-272b-4be0-882e-5d89f2b5223a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:19:05.079567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:05.079529    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e9c97df-e472-41ac-b9a8-6cda15da8dd4" path="/var/lib/kubelet/pods/6e9c97df-e472-41ac-b9a8-6cda15da8dd4/volumes"
Oct 30 21:19:07.380243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:07.380214    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"]
Oct 30 21:19:07.384436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:07.384411    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-lllwl"]
Oct 30 21:19:08.483712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.483689    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"
Oct 30 21:19:08.518523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.518500    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"
Oct 30 21:19:08.556881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.556850    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad901042-4981-479b-98a3-8c6b08db50a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad901042-4981-479b-98a3-8c6b08db50a2-plugins-config-volume\") pod \"ad901042-4981-479b-98a3-8c6b08db50a2\" (UID: \"ad901042-4981-479b-98a3-8c6b08db50a2\") "
Oct 30 21:19:08.557071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.556943    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1495e8dd-272b-4be0-882e-5d89f2b5223a-kube-api-access-mh8lf\" (UniqueName: \"kubernetes.io/projected/1495e8dd-272b-4be0-882e-5d89f2b5223a-kube-api-access-mh8lf\") pod \"1495e8dd-272b-4be0-882e-5d89f2b5223a\" (UID: \"1495e8dd-272b-4be0-882e-5d89f2b5223a\") "
Oct 30 21:19:08.557071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.557053    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad901042-4981-479b-98a3-8c6b08db50a2-kube-api-access-bqm82\" (UniqueName: \"kubernetes.io/projected/ad901042-4981-479b-98a3-8c6b08db50a2-kube-api-access-bqm82\") pod \"ad901042-4981-479b-98a3-8c6b08db50a2\" (UID: \"ad901042-4981-479b-98a3-8c6b08db50a2\") "
Oct 30 21:19:08.557244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.557223    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1495e8dd-272b-4be0-882e-5d89f2b5223a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1495e8dd-272b-4be0-882e-5d89f2b5223a-plugins-config-volume\") pod \"1495e8dd-272b-4be0-882e-5d89f2b5223a\" (UID: \"1495e8dd-272b-4be0-882e-5d89f2b5223a\") "
Oct 30 21:19:08.557310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.557289    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad901042-4981-479b-98a3-8c6b08db50a2-plugins-config-volume" pod "ad901042-4981-479b-98a3-8c6b08db50a2" (UID: "ad901042-4981-479b-98a3-8c6b08db50a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:08.557583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.557562    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1495e8dd-272b-4be0-882e-5d89f2b5223a-plugins-config-volume" pod "1495e8dd-272b-4be0-882e-5d89f2b5223a" (UID: "1495e8dd-272b-4be0-882e-5d89f2b5223a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:08.557803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.557786    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad901042-4981-479b-98a3-8c6b08db50a2-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:08.557899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.557805    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1495e8dd-272b-4be0-882e-5d89f2b5223a-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:08.559453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.559380    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1495e8dd-272b-4be0-882e-5d89f2b5223a-kube-api-access-mh8lf" pod "1495e8dd-272b-4be0-882e-5d89f2b5223a" (UID: "1495e8dd-272b-4be0-882e-5d89f2b5223a"). InnerVolumeSpecName "kube-api-access-mh8lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:08.559783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.559766    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad901042-4981-479b-98a3-8c6b08db50a2-kube-api-access-bqm82" pod "ad901042-4981-479b-98a3-8c6b08db50a2" (UID: "ad901042-4981-479b-98a3-8c6b08db50a2"). InnerVolumeSpecName "kube-api-access-bqm82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:08.603412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.603362    4918 generic.go:356] "Generic (PLEG): container finished" podID="1495e8dd-272b-4be0-882e-5d89f2b5223a" containerID="11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5" exitCode=0
Oct 30 21:19:08.603597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.603438    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"
Oct 30 21:19:08.603597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.603442    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb" event={"ID":"1495e8dd-272b-4be0-882e-5d89f2b5223a","Type":"ContainerDied","Data":"11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5"}
Oct 30 21:19:08.603597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.603497    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb" event={"ID":"1495e8dd-272b-4be0-882e-5d89f2b5223a","Type":"ContainerDied","Data":"25170fd45914144d2464bfd3ff52ab67aaee88148d61784f099bef865d21a027"}
Oct 30 21:19:08.603597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.603524    4918 scope.go:122] "RemoveContainer" containerID="11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5"
Oct 30 21:19:08.604990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.604973    4918 generic.go:356] "Generic (PLEG): container finished" podID="ad901042-4981-479b-98a3-8c6b08db50a2" containerID="20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca" exitCode=0
Oct 30 21:19:08.605109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.605023    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"
Oct 30 21:19:08.605109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.605066    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf" event={"ID":"ad901042-4981-479b-98a3-8c6b08db50a2","Type":"ContainerDied","Data":"20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca"}
Oct 30 21:19:08.605218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.605123    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf" event={"ID":"ad901042-4981-479b-98a3-8c6b08db50a2","Type":"ContainerDied","Data":"f8234f853dd936e5c1b856d1117c96e489bfbd47ca6964ace0dcfeb1923dc478"}
Oct 30 21:19:08.615188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.615169    4918 scope.go:122] "RemoveContainer" containerID="11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5"
Oct 30 21:19:08.615508 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:19:08.615490    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5\": container with ID starting with 11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5 not found: ID does not exist" containerID="11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5"
Oct 30 21:19:08.615562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.615518    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5"} err="failed to get container status \"11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5\": rpc error: code = NotFound desc = could not find container \"11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5\": container with ID starting with 11525825879d2c618b5c6e929f05c18afe0323cbd57270589d12cf1f7fdd79b5 not found: ID does not exist"
Oct 30 21:19:08.615562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.615534    4918 scope.go:122] "RemoveContainer" containerID="20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca"
Oct 30 21:19:08.627765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.627746    4918 scope.go:122] "RemoveContainer" containerID="20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca"
Oct 30 21:19:08.628069 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:19:08.628045    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca\": container with ID starting with 20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca not found: ID does not exist" containerID="20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca"
Oct 30 21:19:08.628261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.628083    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca"} err="failed to get container status \"20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca\": rpc error: code = NotFound desc = could not find container \"20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca\": container with ID starting with 20e5746d1f985ea67b5eafec780ae5debd6b01c572d9519dd817e695d3b41fca not found: ID does not exist"
Oct 30 21:19:08.630690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.630667    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"]
Oct 30 21:19:08.635130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.635110    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-kdhdb"]
Oct 30 21:19:08.645087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.645064    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"]
Oct 30 21:19:08.650994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.650971    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-d9wcf"]
Oct 30 21:19:08.658884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.658865    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh8lf\" (UniqueName: \"kubernetes.io/projected/1495e8dd-272b-4be0-882e-5d89f2b5223a-kube-api-access-mh8lf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:08.658884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:08.658886    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqm82\" (UniqueName: \"kubernetes.io/projected/ad901042-4981-479b-98a3-8c6b08db50a2-kube-api-access-bqm82\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:09.077630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:09.077598    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1495e8dd-272b-4be0-882e-5d89f2b5223a" path="/var/lib/kubelet/pods/1495e8dd-272b-4be0-882e-5d89f2b5223a/volumes"
Oct 30 21:19:09.078011 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:09.077997    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad901042-4981-479b-98a3-8c6b08db50a2" path="/var/lib/kubelet/pods/ad901042-4981-479b-98a3-8c6b08db50a2/volumes"
Oct 30 21:19:09.078434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:09.078416    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b6b17c-0b3d-4e19-a252-e15dd220681d" path="/var/lib/kubelet/pods/b9b6b17c-0b3d-4e19-a252-e15dd220681d/volumes"
Oct 30 21:19:11.020180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:11.020144    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"]
Oct 30 21:19:11.021425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:11.020431    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy" containerID="cri-o://30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d" gracePeriod=30
Oct 30 21:19:12.204125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.204094    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"]
Oct 30 21:19:12.204712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.204699    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1495e8dd-272b-4be0-882e-5d89f2b5223a" containerName="payload-processing"
Oct 30 21:19:12.204771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.204713    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1495e8dd-272b-4be0-882e-5d89f2b5223a" containerName="payload-processing"
Oct 30 21:19:12.204771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.204752    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ad901042-4981-479b-98a3-8c6b08db50a2" containerName="payload-pre-processing"
Oct 30 21:19:12.204771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.204757    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad901042-4981-479b-98a3-8c6b08db50a2" containerName="payload-pre-processing"
Oct 30 21:19:12.204866 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.204848    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1495e8dd-272b-4be0-882e-5d89f2b5223a" containerName="payload-processing"
Oct 30 21:19:12.204904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.204866    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad901042-4981-479b-98a3-8c6b08db50a2" containerName="payload-pre-processing"
Oct 30 21:19:12.210199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.210179    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:12.259682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.259656    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:19:12.259955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.259935    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:19:12.304520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.304491    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"]
Oct 30 21:19:12.307760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.307728    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:19:12.308013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.307984    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:19:12.322812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.322790    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5518aeae-dgtd9\" (UID: \"624ca571-f68a-45e4-8d12-8a0378147e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:12.323017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.323001    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5518aeae-dgtd9\" (UID: \"624ca571-f68a-45e4-8d12-8a0378147e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:12.323078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.323062    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw4dq\" (UniqueName: \"kubernetes.io/projected/624ca571-f68a-45e4-8d12-8a0378147e94-kube-api-access-bw4dq\") pod \"maas-api-revoke-sub-5518aeae-dgtd9\" (UID: \"624ca571-f68a-45e4-8d12-8a0378147e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:12.424072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.424042    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5518aeae-dgtd9\" (UID: \"624ca571-f68a-45e4-8d12-8a0378147e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:12.424253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.424114    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw4dq\" (UniqueName: \"kubernetes.io/projected/624ca571-f68a-45e4-8d12-8a0378147e94-kube-api-access-bw4dq\") pod \"maas-api-revoke-sub-5518aeae-dgtd9\" (UID: \"624ca571-f68a-45e4-8d12-8a0378147e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:12.424253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.424165    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5518aeae-dgtd9\" (UID: \"624ca571-f68a-45e4-8d12-8a0378147e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:12.424733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.424711    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5518aeae-dgtd9\" (UID: \"624ca571-f68a-45e4-8d12-8a0378147e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:12.426805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.426784    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5518aeae-dgtd9\" (UID: \"624ca571-f68a-45e4-8d12-8a0378147e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:12.433727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.433709    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw4dq\" (UniqueName: \"kubernetes.io/projected/624ca571-f68a-45e4-8d12-8a0378147e94-kube-api-access-bw4dq\") pod \"maas-api-revoke-sub-5518aeae-dgtd9\" (UID: \"624ca571-f68a-45e4-8d12-8a0378147e94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:12.525938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.525907    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:12.810284 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:19:12.810255    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624ca571_f68a_45e4_8d12_8a0378147e94.slice/crio-4c7db04d84d16399c2b2dc574ba35063f6af1fa35a811d324646733f53f35417 WatchSource:0}: Error finding container 4c7db04d84d16399c2b2dc574ba35063f6af1fa35a811d324646733f53f35417: Status 404 returned error can't find the container with id 4c7db04d84d16399c2b2dc574ba35063f6af1fa35a811d324646733f53f35417
Oct 30 21:19:12.810419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:12.810281    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"]
Oct 30 21:19:13.738354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:13.738320    4918 generic.go:356] "Generic (PLEG): container finished" podID="624ca571-f68a-45e4-8d12-8a0378147e94" containerID="bd0843013f28fd895a5c7d815481a71018c687cc973dc912a0130885c22954b0" exitCode=0
Oct 30 21:19:13.738704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:13.738367    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9" event={"ID":"624ca571-f68a-45e4-8d12-8a0378147e94","Type":"ContainerDied","Data":"bd0843013f28fd895a5c7d815481a71018c687cc973dc912a0130885c22954b0"}
Oct 30 21:19:13.738704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:13.738461    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9" event={"ID":"624ca571-f68a-45e4-8d12-8a0378147e94","Type":"ContainerStarted","Data":"4c7db04d84d16399c2b2dc574ba35063f6af1fa35a811d324646733f53f35417"}
Oct 30 21:19:15.016573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.016551    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:15.112224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.112144    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-openshift-ca\") pod \"624ca571-f68a-45e4-8d12-8a0378147e94\" (UID: \"624ca571-f68a-45e4-8d12-8a0378147e94\") "
Oct 30 21:19:15.112411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.112310    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-xks-ca\") pod \"624ca571-f68a-45e4-8d12-8a0378147e94\" (UID: \"624ca571-f68a-45e4-8d12-8a0378147e94\") "
Oct 30 21:19:15.112536 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.112521    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/624ca571-f68a-45e4-8d12-8a0378147e94-kube-api-access-bw4dq\" (UniqueName: \"kubernetes.io/projected/624ca571-f68a-45e4-8d12-8a0378147e94-kube-api-access-bw4dq\") pod \"624ca571-f68a-45e4-8d12-8a0378147e94\" (UID: \"624ca571-f68a-45e4-8d12-8a0378147e94\") "
Oct 30 21:19:15.112598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.112524    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-openshift-ca" pod "624ca571-f68a-45e4-8d12-8a0378147e94" (UID: "624ca571-f68a-45e4-8d12-8a0378147e94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:15.112968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.112949    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:15.114493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.114470    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-xks-ca" pod "624ca571-f68a-45e4-8d12-8a0378147e94" (UID: "624ca571-f68a-45e4-8d12-8a0378147e94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:15.114667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.114642    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624ca571-f68a-45e4-8d12-8a0378147e94-kube-api-access-bw4dq" pod "624ca571-f68a-45e4-8d12-8a0378147e94" (UID: "624ca571-f68a-45e4-8d12-8a0378147e94"). InnerVolumeSpecName "kube-api-access-bw4dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:15.214585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.214555    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/624ca571-f68a-45e4-8d12-8a0378147e94-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:15.214585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.214581    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw4dq\" (UniqueName: \"kubernetes.io/projected/624ca571-f68a-45e4-8d12-8a0378147e94-kube-api-access-bw4dq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:15.802566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.802536    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"
Oct 30 21:19:15.802767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.802574    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9" event={"ID":"624ca571-f68a-45e4-8d12-8a0378147e94","Type":"ContainerDied","Data":"4c7db04d84d16399c2b2dc574ba35063f6af1fa35a811d324646733f53f35417"}
Oct 30 21:19:15.802767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:15.802623    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c7db04d84d16399c2b2dc574ba35063f6af1fa35a811d324646733f53f35417"
Oct 30 21:19:16.402201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.402135    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:19:16.437754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.437721    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-credential-socket\") pod \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") "
Oct 30 21:19:16.437940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.437778    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-kube-api-access-rtkfk\" (UniqueName: \"kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-kube-api-access-rtkfk\") pod \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") "
Oct 30 21:19:16.437940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.437807    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-socket\") pod \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") "
Oct 30 21:19:16.437940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.437853    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-data\") pod \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") "
Oct 30 21:19:16.437940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.437930    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-envoy\") pod \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") "
Oct 30 21:19:16.438146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.437974    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-token\" (UniqueName: \"kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-token\") pod \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") "
Oct 30 21:19:16.438146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.438045    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-credential-socket" pod "6ae7a5a7-5cd7-4094-863c-2de252eda90b" (UID: "6ae7a5a7-5cd7-4094-863c-2de252eda90b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:16.438146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.438136    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-certs\") pod \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") "
Oct 30 21:19:16.438313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.438225    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-data" pod "6ae7a5a7-5cd7-4094-863c-2de252eda90b" (UID: "6ae7a5a7-5cd7-4094-863c-2de252eda90b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:16.438313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.438265    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istiod-ca-cert\") pod \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") "
Oct 30 21:19:16.438476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.438343    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-podinfo\") pod \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\" (UID: \"6ae7a5a7-5cd7-4094-863c-2de252eda90b\") "
Oct 30 21:19:16.438541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.438502    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-certs" pod "6ae7a5a7-5cd7-4094-863c-2de252eda90b" (UID: "6ae7a5a7-5cd7-4094-863c-2de252eda90b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:16.438809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.438788    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istiod-ca-cert" pod "6ae7a5a7-5cd7-4094-863c-2de252eda90b" (UID: "6ae7a5a7-5cd7-4094-863c-2de252eda90b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:16.439040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.439014    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-socket" pod "6ae7a5a7-5cd7-4094-863c-2de252eda90b" (UID: "6ae7a5a7-5cd7-4094-863c-2de252eda90b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:16.439356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.439296    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:16.439356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.439318    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:16.439356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.439333    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:16.439356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.439347    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:16.439356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.439360    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:16.440582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.440554    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-token" pod "6ae7a5a7-5cd7-4094-863c-2de252eda90b" (UID: "6ae7a5a7-5cd7-4094-863c-2de252eda90b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:16.440818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.440799    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-envoy" pod "6ae7a5a7-5cd7-4094-863c-2de252eda90b" (UID: "6ae7a5a7-5cd7-4094-863c-2de252eda90b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:19:16.441016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.440989    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-kube-api-access-rtkfk" pod "6ae7a5a7-5cd7-4094-863c-2de252eda90b" (UID: "6ae7a5a7-5cd7-4094-863c-2de252eda90b"). InnerVolumeSpecName "kube-api-access-rtkfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:16.441098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.441080    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-podinfo" pod "6ae7a5a7-5cd7-4094-863c-2de252eda90b" (UID: "6ae7a5a7-5cd7-4094-863c-2de252eda90b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:19:16.541196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.541162    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:16.541196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.541189    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtkfk\" (UniqueName: \"kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-kube-api-access-rtkfk\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:16.541196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.541200    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:16.541451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.541210    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae7a5a7-5cd7-4094-863c-2de252eda90b-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:16.609990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.609950    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"]
Oct 30 21:19:16.610653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.610634    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy"
Oct 30 21:19:16.610653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.610651    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy"
Oct 30 21:19:16.610884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.610662    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy"
Oct 30 21:19:16.610884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.610668    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy"
Oct 30 21:19:16.610884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.610682    4918 cpu_manager.go:414] "containerMap: removing container" podUID="624ca571-f68a-45e4-8d12-8a0378147e94" containerName="revoke-keys"
Oct 30 21:19:16.610884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.610688    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624ca571-f68a-45e4-8d12-8a0378147e94" containerName="revoke-keys"
Oct 30 21:19:16.610884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.610777    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy"
Oct 30 21:19:16.610884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.610789    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerName="istio-proxy"
Oct 30 21:19:16.610884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.610800    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="624ca571-f68a-45e4-8d12-8a0378147e94" containerName="revoke-keys"
Oct 30 21:19:16.614878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.614858    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.621931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.621897    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4c6998-openshift-default-dockercfg-dpkjz\""
Oct 30 21:19:16.628925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.628896    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"]
Oct 30 21:19:16.642228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.642192    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-certs\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.642384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.642294    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bkhh\" (UniqueName: \"kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-kube-api-access-2bkhh\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.642384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.642328    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-socket\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.642384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.642359    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istiod-ca-cert\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.642594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.642526    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-token\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.642594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.642555    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-data\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.642705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.642662    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-credential-socket\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.642933 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.642902    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-podinfo\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.643063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.642954    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-envoy\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.750099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.749852    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-podinfo\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.750305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.750121    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-envoy\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.750305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.750152    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-certs\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.750305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.750291    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bkhh\" (UniqueName: \"kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-kube-api-access-2bkhh\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.750491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.750336    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-socket\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.750491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.750402    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istiod-ca-cert\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.750491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.750474    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-token\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.750491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.750503    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-data\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.750724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.750607    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-credential-socket\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.750874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.750833    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-certs\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.751036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.751017    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-data\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.751332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.751302    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istiod-ca-cert\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.751662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.751619    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-socket\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.752048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.752026    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-credential-socket\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.753135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.753087    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-podinfo\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.754592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.754549    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-envoy\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.877056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.877019    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bkhh\" (UniqueName: \"kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-kube-api-access-2bkhh\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.901605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.901572    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-token\") pod \"e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") " pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:16.914092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.914056    4918 generic.go:356] "Generic (PLEG): container finished" podID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" containerID="30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d" exitCode=0
Oct 30 21:19:16.914549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.914523    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" event={"ID":"6ae7a5a7-5cd7-4094-863c-2de252eda90b","Type":"ContainerDied","Data":"30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d"}
Oct 30 21:19:16.914773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.914757    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf" event={"ID":"6ae7a5a7-5cd7-4094-863c-2de252eda90b","Type":"ContainerDied","Data":"3879b70e9f9d4bae5176f73cbfb2a7243e7aeb3864939af251ae1b5693d41aa1"}
Oct 30 21:19:16.914892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.914881    4918 scope.go:122] "RemoveContainer" containerID="30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d"
Oct 30 21:19:16.915180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.915162    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"
Oct 30 21:19:16.949766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:16.949740    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:17.004520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.001585    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"]
Oct 30 21:19:17.010701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.008956    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b3d1de-openshift-default-785fc5bd85-q4dhf"]
Oct 30 21:19:17.053978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.043357    4918 scope.go:122] "RemoveContainer" containerID="5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec"
Oct 30 21:19:17.084068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.084037    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae7a5a7-5cd7-4094-863c-2de252eda90b" path="/var/lib/kubelet/pods/6ae7a5a7-5cd7-4094-863c-2de252eda90b/volumes"
Oct 30 21:19:17.091696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.091667    4918 scope.go:122] "RemoveContainer" containerID="30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d"
Oct 30 21:19:17.092074 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:19:17.092056    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d\": container with ID starting with 30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d not found: ID does not exist" containerID="30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d"
Oct 30 21:19:17.092162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.092087    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d"} err="failed to get container status \"30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d\": rpc error: code = NotFound desc = could not find container \"30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d\": container with ID starting with 30a31d7b46da9dac67e9b98b90d85b3398ab474013720b1457d9508480819c5d not found: ID does not exist"
Oct 30 21:19:17.092162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.092107    4918 scope.go:122] "RemoveContainer" containerID="5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec"
Oct 30 21:19:17.092382 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:19:17.092367    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec\": container with ID starting with 5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec not found: ID does not exist" containerID="5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec"
Oct 30 21:19:17.092460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.092426    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec"} err="failed to get container status \"5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec\": rpc error: code = NotFound desc = could not find container \"5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec\": container with ID starting with 5b6b9fc5e141e18b2171d355ec9cf38cf62d0baa12049397cd4293c28072e2ec not found: ID does not exist"
Oct 30 21:19:17.354766 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:19:17.354739    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ab1dc4_eb49_4815_8c11_00f9cf59602a.slice/crio-62121475619b16fd30bdf858a1e05c4685e4a20be04bcd47f7609b55d2fd09ca WatchSource:0}: Error finding container 62121475619b16fd30bdf858a1e05c4685e4a20be04bcd47f7609b55d2fd09ca: Status 404 returned error can't find the container with id 62121475619b16fd30bdf858a1e05c4685e4a20be04bcd47f7609b55d2fd09ca
Oct 30 21:19:17.355907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.355883    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"]
Oct 30 21:19:17.358044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.358004    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:19:17.358119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.358092    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:19:17.358159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.358136    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:19:17.925558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.925526    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" event={"ID":"81ab1dc4-eb49-4815-8c11-00f9cf59602a","Type":"ContainerStarted","Data":"dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa"}
Oct 30 21:19:17.926021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.925581    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" event={"ID":"81ab1dc4-eb49-4815-8c11-00f9cf59602a","Type":"ContainerStarted","Data":"62121475619b16fd30bdf858a1e05c4685e4a20be04bcd47f7609b55d2fd09ca"}
Oct 30 21:19:17.950018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.949972    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" podStartSLOduration=1.9499585910000001 podStartE2EDuration="1.949958591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:17.948124939 +0000 UTC m=+1070.994684098" watchObservedRunningTime="2026-10-30 21:19:17.949958591 +0000 UTC m=+1070.996517735"
Oct 30 21:19:17.950318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:17.950292    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:18.958343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:18.958317    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:19.301648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.301618    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"]
Oct 30 21:19:19.306206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.306188    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"
Oct 30 21:19:19.319629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.319601    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5bc8c60a\""
Oct 30 21:19:19.334896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.334804    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"]
Oct 30 21:19:19.343728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.343697    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"]
Oct 30 21:19:19.349570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.349546    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"
Oct 30 21:19:19.355146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.355123    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"]
Oct 30 21:19:19.355505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.355479    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5bc8c60a-dockercfg-l6dl7\""
Oct 30 21:19:19.408668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.408631    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2xvf\" (UniqueName: \"kubernetes.io/projected/af040445-e395-46fd-9e3d-a617d0d6ad94-kube-api-access-z2xvf\") pod \"payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4\" (UID: \"af040445-e395-46fd-9e3d-a617d0d6ad94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"
Oct 30 21:19:19.408870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.408800    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af040445-e395-46fd-9e3d-a617d0d6ad94-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4\" (UID: \"af040445-e395-46fd-9e3d-a617d0d6ad94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"
Oct 30 21:19:19.517417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.514441    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af040445-e395-46fd-9e3d-a617d0d6ad94-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4\" (UID: \"af040445-e395-46fd-9e3d-a617d0d6ad94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"
Oct 30 21:19:19.517417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.514581    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79p4p\" (UniqueName: \"kubernetes.io/projected/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-kube-api-access-79p4p\") pod \"payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x\" (UID: \"1f2c2d6b-cd9f-4c54-86d2-d897d3056800\") " pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"
Oct 30 21:19:19.517417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.514667    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-plugins-config-volume\") pod \"payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x\" (UID: \"1f2c2d6b-cd9f-4c54-86d2-d897d3056800\") " pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"
Oct 30 21:19:19.517417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.514760    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2xvf\" (UniqueName: \"kubernetes.io/projected/af040445-e395-46fd-9e3d-a617d0d6ad94-kube-api-access-z2xvf\") pod \"payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4\" (UID: \"af040445-e395-46fd-9e3d-a617d0d6ad94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"
Oct 30 21:19:19.517417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.516019    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af040445-e395-46fd-9e3d-a617d0d6ad94-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4\" (UID: \"af040445-e395-46fd-9e3d-a617d0d6ad94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"
Oct 30 21:19:19.568688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.567113    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2xvf\" (UniqueName: \"kubernetes.io/projected/af040445-e395-46fd-9e3d-a617d0d6ad94-kube-api-access-z2xvf\") pod \"payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4\" (UID: \"af040445-e395-46fd-9e3d-a617d0d6ad94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"
Oct 30 21:19:19.622434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.620786    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"
Oct 30 21:19:19.622434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.621590    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79p4p\" (UniqueName: \"kubernetes.io/projected/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-kube-api-access-79p4p\") pod \"payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x\" (UID: \"1f2c2d6b-cd9f-4c54-86d2-d897d3056800\") " pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"
Oct 30 21:19:19.622434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.621692    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-plugins-config-volume\") pod \"payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x\" (UID: \"1f2c2d6b-cd9f-4c54-86d2-d897d3056800\") " pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"
Oct 30 21:19:19.623467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.622753    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-plugins-config-volume\") pod \"payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x\" (UID: \"1f2c2d6b-cd9f-4c54-86d2-d897d3056800\") " pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"
Oct 30 21:19:19.670414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.660988    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79p4p\" (UniqueName: \"kubernetes.io/projected/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-kube-api-access-79p4p\") pod \"payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x\" (UID: \"1f2c2d6b-cd9f-4c54-86d2-d897d3056800\") " pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"
Oct 30 21:19:19.682767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.672414    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"
Oct 30 21:19:19.942436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.942380    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:19.943370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:19.943353    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:20.062379 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:19:20.061764    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf040445_e395_46fd_9e3d_a617d0d6ad94.slice/crio-8df7152f528c04c95a26ef79840f344b89ccb02f27c7dd6334ba1e60d17ee68b WatchSource:0}: Error finding container 8df7152f528c04c95a26ef79840f344b89ccb02f27c7dd6334ba1e60d17ee68b: Status 404 returned error can't find the container with id 8df7152f528c04c95a26ef79840f344b89ccb02f27c7dd6334ba1e60d17ee68b
Oct 30 21:19:20.068776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:20.068744    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"]
Oct 30 21:19:20.109787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:20.099718    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"]
Oct 30 21:19:20.117597 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:19:20.113337    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2c2d6b_cd9f_4c54_86d2_d897d3056800.slice/crio-22017dec47dcd6c385d7ced81a861cf4493903d7fb2ba0551b20afa57d21d8fd WatchSource:0}: Error finding container 22017dec47dcd6c385d7ced81a861cf4493903d7fb2ba0551b20afa57d21d8fd: Status 404 returned error can't find the container with id 22017dec47dcd6c385d7ced81a861cf4493903d7fb2ba0551b20afa57d21d8fd
Oct 30 21:19:21.107539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.107476    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4" podStartSLOduration=2.107458046 podStartE2EDuration="2.107458046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:21.104710592 +0000 UTC m=+1074.151269747" watchObservedRunningTime="2026-10-30 21:19:21.107458046 +0000 UTC m=+1074.154017193"
Oct 30 21:19:21.110194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.110156    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4" event={"ID":"af040445-e395-46fd-9e3d-a617d0d6ad94","Type":"ContainerStarted","Data":"a9163864c8a5af653c57fa06d7cdd8d45221d6fc1f8d0c93ffbd3b0e8f765aa2"}
Oct 30 21:19:21.110326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.110225    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"
Oct 30 21:19:21.110326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.110255    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"
Oct 30 21:19:21.110326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.110280    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4" event={"ID":"af040445-e395-46fd-9e3d-a617d0d6ad94","Type":"ContainerStarted","Data":"8df7152f528c04c95a26ef79840f344b89ccb02f27c7dd6334ba1e60d17ee68b"}
Oct 30 21:19:21.110326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.110299    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x" event={"ID":"1f2c2d6b-cd9f-4c54-86d2-d897d3056800","Type":"ContainerStarted","Data":"7cc0a00b3ce524316d7764f4027ec09c2417931a675951b5047c8d8b6607346d"}
Oct 30 21:19:21.110567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.110333    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x" event={"ID":"1f2c2d6b-cd9f-4c54-86d2-d897d3056800","Type":"ContainerStarted","Data":"22017dec47dcd6c385d7ced81a861cf4493903d7fb2ba0551b20afa57d21d8fd"}
Oct 30 21:19:21.128001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.127945    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x" podStartSLOduration=2.127930665 podStartE2EDuration="2.127930665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:21.125075434 +0000 UTC m=+1074.171634582" watchObservedRunningTime="2026-10-30 21:19:21.127930665 +0000 UTC m=+1074.174489810"
Oct 30 21:19:21.641775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.641741    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"]
Oct 30 21:19:21.647222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.647198    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"
Oct 30 21:19:21.654364 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.654337    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"]
Oct 30 21:19:21.837414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.837200    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4c6998\""
Oct 30 21:19:21.837414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.837355    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4c6998-dockercfg-bd7fq\""
Oct 30 21:19:21.891415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.883264    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz\" (UID: \"533db7d6-d52a-4bec-a02d-9fab5ca8db9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"
Oct 30 21:19:21.891415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.883618    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmj5c\" (UniqueName: \"kubernetes.io/projected/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-kube-api-access-jmj5c\") pod \"payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz\" (UID: \"533db7d6-d52a-4bec-a02d-9fab5ca8db9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"
Oct 30 21:19:21.985424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.985347    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmj5c\" (UniqueName: \"kubernetes.io/projected/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-kube-api-access-jmj5c\") pod \"payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz\" (UID: \"533db7d6-d52a-4bec-a02d-9fab5ca8db9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"
Oct 30 21:19:21.985836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.985798    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz\" (UID: \"533db7d6-d52a-4bec-a02d-9fab5ca8db9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"
Oct 30 21:19:21.987142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.987041    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz\" (UID: \"533db7d6-d52a-4bec-a02d-9fab5ca8db9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"
Oct 30 21:19:21.998490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:21.998473    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmj5c\" (UniqueName: \"kubernetes.io/projected/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-kube-api-access-jmj5c\") pod \"payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz\" (UID: \"533db7d6-d52a-4bec-a02d-9fab5ca8db9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"
Oct 30 21:19:22.268914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:22.268823    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"
Oct 30 21:19:22.552450 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:19:22.552419    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533db7d6_d52a_4bec_a02d_9fab5ca8db9b.slice/crio-1b1122f73814f2ed9a9dd6093cbcdb78bed6bd61b45d3d5b25a57fafdec8b935 WatchSource:0}: Error finding container 1b1122f73814f2ed9a9dd6093cbcdb78bed6bd61b45d3d5b25a57fafdec8b935: Status 404 returned error can't find the container with id 1b1122f73814f2ed9a9dd6093cbcdb78bed6bd61b45d3d5b25a57fafdec8b935
Oct 30 21:19:22.553317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:22.553295    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"]
Oct 30 21:19:23.103093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:23.103058    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz" event={"ID":"533db7d6-d52a-4bec-a02d-9fab5ca8db9b","Type":"ContainerStarted","Data":"98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f"}
Oct 30 21:19:23.103261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:23.103108    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz" event={"ID":"533db7d6-d52a-4bec-a02d-9fab5ca8db9b","Type":"ContainerStarted","Data":"1b1122f73814f2ed9a9dd6093cbcdb78bed6bd61b45d3d5b25a57fafdec8b935"}
Oct 30 21:19:23.103261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:23.103162    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"
Oct 30 21:19:23.124162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:23.124110    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz" podStartSLOduration=2.124097513 podStartE2EDuration="2.124097513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:23.120156522 +0000 UTC m=+1076.166715664" watchObservedRunningTime="2026-10-30 21:19:23.124097513 +0000 UTC m=+1076.170656683"
Oct 30 21:19:24.825209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.825173    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"]
Oct 30 21:19:24.837920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.837889    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"]
Oct 30 21:19:24.838299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.838280    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:24.845874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.845806    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3ac6bf-openshift-default-dockercfg-vznzc\""
Oct 30 21:19:24.960775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.960740    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:24.960975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.960818    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-data\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:24.960975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.960910    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a16952-efea-4b33-a112-c4b533c353c9-istiod-ca-cert\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:24.961107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.961015    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-credential-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:24.961107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.961090    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-certs\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:24.961223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.961128    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a16952-efea-4b33-a112-c4b533c353c9-istio-podinfo\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:24.961286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.961239    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbs8q\" (UniqueName: \"kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-kube-api-access-fbs8q\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:24.961350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.961303    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-istio-token\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:24.961453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:24.961358    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-envoy\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.066411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.063373    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-credential-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.066411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.063496    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-certs\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.066411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.063544    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a16952-efea-4b33-a112-c4b533c353c9-istio-podinfo\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.066411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.063627    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbs8q\" (UniqueName: \"kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-kube-api-access-fbs8q\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.066411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.063673    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-istio-token\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.066411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.063731    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-envoy\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.066411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.063861    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.066411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.063884    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-data\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.066411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.063937    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a16952-efea-4b33-a112-c4b533c353c9-istiod-ca-cert\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.066411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.064809    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a16952-efea-4b33-a112-c4b533c353c9-istiod-ca-cert\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.066411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.065313    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-credential-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.066411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.065684    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-certs\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.070442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.069081    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-socket\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.070442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.069419    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-data\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.149476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.140899    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-envoy\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.173414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.151125    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbs8q\" (UniqueName: \"kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-kube-api-access-fbs8q\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.173414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.151170    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-istio-token\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.188419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.175440    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a16952-efea-4b33-a112-c4b533c353c9-istio-podinfo\") pod \"e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") " pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.483096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.483065    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:25.861722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.861684    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"]
Oct 30 21:19:25.863034 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:19:25.863009    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a16952_efea_4b33_a112_c4b533c353c9.slice/crio-65f293baec6c13ceb15503226b31921465ffa9aa72bee790d347a9e3179459c8 WatchSource:0}: Error finding container 65f293baec6c13ceb15503226b31921465ffa9aa72bee790d347a9e3179459c8: Status 404 returned error can't find the container with id 65f293baec6c13ceb15503226b31921465ffa9aa72bee790d347a9e3179459c8
Oct 30 21:19:25.866613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.866574    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:19:25.866697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.866665    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:19:25.866734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:25.866710    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:19:26.397574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:26.397535    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" event={"ID":"97a16952-efea-4b33-a112-c4b533c353c9","Type":"ContainerStarted","Data":"a51a86fc68a8f6d509a9a9aea97fb6a579cb2f0c2b998464340ccd3461089916"}
Oct 30 21:19:26.397881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:26.397856    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" event={"ID":"97a16952-efea-4b33-a112-c4b533c353c9","Type":"ContainerStarted","Data":"65f293baec6c13ceb15503226b31921465ffa9aa72bee790d347a9e3179459c8"}
Oct 30 21:19:26.473415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:26.454291    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" podStartSLOduration=2.454271346 podStartE2EDuration="2.454271346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:26.452735117 +0000 UTC m=+1079.499294263" watchObservedRunningTime="2026-10-30 21:19:26.454271346 +0000 UTC m=+1079.500830491"
Oct 30 21:19:26.484434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:26.483665    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:26.508666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:26.508630    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 21:19:26.508938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:26.508910    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 21:19:27.060062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.060021    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"]
Oct 30 21:19:27.068176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.068153    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:27.075620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.075593    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:19:27.075776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.075718    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:19:27.076310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.076284    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:19:27.076437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.076363    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:19:27.104432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.098751    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"]
Oct 30 21:19:27.175417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.161673    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcrmw\" (UniqueName: \"kubernetes.io/projected/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-kube-api-access-wcrmw\") pod \"maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w\" (UID: \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:27.175417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.161781    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w\" (UID: \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:27.175417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.161866    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w\" (UID: \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:27.265426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.263317    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w\" (UID: \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:27.265426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.263427    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w\" (UID: \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:27.265426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.263703    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcrmw\" (UniqueName: \"kubernetes.io/projected/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-kube-api-access-wcrmw\") pod \"maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w\" (UID: \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:27.269423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.268540    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:19:27.277423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.276246    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w\" (UID: \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:27.278611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.278595    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:19:27.279574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.279549    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w\" (UID: \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:27.317467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.309250    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcrmw\" (UniqueName: \"kubernetes.io/projected/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-kube-api-access-wcrmw\") pod \"maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w\" (UID: \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:27.465419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.464794    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:27.538750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:27.538716    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:28.083322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:28.083226    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"]
Oct 30 21:19:28.476385 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:28.476348    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w" event={"ID":"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5","Type":"ContainerStarted","Data":"b970c04c438d00fdb9d4fcacf9762f7bdef6a67fa0dd5b3f03e52ffa827d2225"}
Oct 30 21:19:28.476885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:28.476870    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:28.478934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:28.478916    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:19:28.546117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:28.546083    4918 scope.go:122] "RemoveContainer" containerID="e669515a782e0d89c65a3b9549beb8cfb39886fcf9bdf6cd6c5cd9f7f3ad571d"
Oct 30 21:19:28.557699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:28.557667    4918 scope.go:122] "RemoveContainer" containerID="8868a2ae6f98ce1316d9a76c99d405b658d8134905ea2df49d161cd40688d3ba"
Oct 30 21:19:28.569731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:28.569708    4918 scope.go:122] "RemoveContainer" containerID="fd1ec70b0336da6f8da79fb2b6fea11e9ad70c2ea6cc3d82796674a3fe27214e"
Oct 30 21:19:28.588548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:28.588523    4918 scope.go:122] "RemoveContainer" containerID="046f26493012766ca4d959ca46bb88adde0035e0f92c06bd5633d6c9167f4896"
Oct 30 21:19:28.601232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:28.601213    4918 scope.go:122] "RemoveContainer" containerID="82ee1a63a770912fe4cedadc9d48ccc29f7a053d76d0a5a9f42dc1a723ff127c"
Oct 30 21:19:29.160148 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:19:29.160109    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:19:29.485560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:29.485530    4918 generic.go:356] "Generic (PLEG): container finished" podID="9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5" containerID="f59673a9befabe5b24a47c03cc7ac80b34a54ae303614ec3dedc225007142887" exitCode=0
Oct 30 21:19:29.485767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:29.485615    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w" event={"ID":"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5","Type":"ContainerDied","Data":"f59673a9befabe5b24a47c03cc7ac80b34a54ae303614ec3dedc225007142887"}
Oct 30 21:19:30.159173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.159141    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"]
Oct 30 21:19:30.163724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.163706    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"
Oct 30 21:19:30.172438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.172179    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3ac6bf-dockercfg-vnxdv\""
Oct 30 21:19:30.174190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.174151    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3ac6bf\""
Oct 30 21:19:30.176370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.176348    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"]
Oct 30 21:19:30.257267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.257230    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513ace8b-991a-4516-a054-20e8b568f6e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh\" (UID: \"513ace8b-991a-4516-a054-20e8b568f6e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"
Oct 30 21:19:30.257499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.257292    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg86k\" (UniqueName: \"kubernetes.io/projected/513ace8b-991a-4516-a054-20e8b568f6e3-kube-api-access-bg86k\") pod \"payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh\" (UID: \"513ace8b-991a-4516-a054-20e8b568f6e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"
Oct 30 21:19:30.361438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.361383    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513ace8b-991a-4516-a054-20e8b568f6e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh\" (UID: \"513ace8b-991a-4516-a054-20e8b568f6e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"
Oct 30 21:19:30.362669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.362643    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg86k\" (UniqueName: \"kubernetes.io/projected/513ace8b-991a-4516-a054-20e8b568f6e3-kube-api-access-bg86k\") pod \"payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh\" (UID: \"513ace8b-991a-4516-a054-20e8b568f6e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"
Oct 30 21:19:30.367883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.362551    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513ace8b-991a-4516-a054-20e8b568f6e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh\" (UID: \"513ace8b-991a-4516-a054-20e8b568f6e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"
Oct 30 21:19:30.440050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.421918    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg86k\" (UniqueName: \"kubernetes.io/projected/513ace8b-991a-4516-a054-20e8b568f6e3-kube-api-access-bg86k\") pod \"payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh\" (UID: \"513ace8b-991a-4516-a054-20e8b568f6e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"
Oct 30 21:19:30.481965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.481935    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"
Oct 30 21:19:30.827036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.827011    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:30.885927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.885896    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-xks-ca\") pod \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\" (UID: \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\") "
Oct 30 21:19:30.886132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.886112    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-openshift-ca\") pod \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\" (UID: \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\") "
Oct 30 21:19:30.886303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.886289    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-kube-api-access-wcrmw\" (UniqueName: \"kubernetes.io/projected/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-kube-api-access-wcrmw\") pod \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\" (UID: \"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5\") "
Oct 30 21:19:30.886523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.886496    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-openshift-ca" pod "9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5" (UID: "9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:30.886851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.886832    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:30.888136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.888115    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-xks-ca" pod "9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5" (UID: "9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:30.888371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.888352    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-kube-api-access-wcrmw" pod "9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5" (UID: "9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5"). InnerVolumeSpecName "kube-api-access-wcrmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:30.988489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.988408    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcrmw\" (UniqueName: \"kubernetes.io/projected/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-kube-api-access-wcrmw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:30.988489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:30.988439    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:31.065263 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:19:31.065235    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513ace8b_991a_4516_a054_20e8b568f6e3.slice/crio-4756460a513b3b7f7be7dfa7dcb5307d949701f15617e4ffc3884d7f6670ecb5 WatchSource:0}: Error finding container 4756460a513b3b7f7be7dfa7dcb5307d949701f15617e4ffc3884d7f6670ecb5: Status 404 returned error can't find the container with id 4756460a513b3b7f7be7dfa7dcb5307d949701f15617e4ffc3884d7f6670ecb5
Oct 30 21:19:31.065419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:31.065380    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"]
Oct 30 21:19:31.541970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:31.541941    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"
Oct 30 21:19:31.542355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:31.541966    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w" event={"ID":"9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5","Type":"ContainerDied","Data":"b970c04c438d00fdb9d4fcacf9762f7bdef6a67fa0dd5b3f03e52ffa827d2225"}
Oct 30 21:19:31.542355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:31.542011    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b970c04c438d00fdb9d4fcacf9762f7bdef6a67fa0dd5b3f03e52ffa827d2225"
Oct 30 21:19:31.544616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:31.544578    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh" event={"ID":"513ace8b-991a-4516-a054-20e8b568f6e3","Type":"ContainerStarted","Data":"bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388"}
Oct 30 21:19:31.544793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:31.544640    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh" event={"ID":"513ace8b-991a-4516-a054-20e8b568f6e3","Type":"ContainerStarted","Data":"4756460a513b3b7f7be7dfa7dcb5307d949701f15617e4ffc3884d7f6670ecb5"}
Oct 30 21:19:31.544905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:31.544884    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"
Oct 30 21:19:31.583412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:31.579029    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh" podStartSLOduration=1.579010588 podStartE2EDuration="1.579010588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:31.573904274 +0000 UTC m=+1084.620463419" watchObservedRunningTime="2026-10-30 21:19:31.579010588 +0000 UTC m=+1084.625569735"
Oct 30 21:19:32.096666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:32.096638    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"
Oct 30 21:19:32.096819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:32.096719    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"
Oct 30 21:19:34.159110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:34.159084    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"
Oct 30 21:19:37.330101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:37.329598    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"]
Oct 30 21:19:41.754971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:41.754928    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7657-2ff7df"]
Oct 30 21:19:41.755852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:41.755830    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5" containerName="revoke-keys"
Oct 30 21:19:41.755970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:41.755853    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5" containerName="revoke-keys"
Oct 30 21:19:41.756065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:41.756050    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5" containerName="revoke-keys"
Oct 30 21:19:41.760663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:41.760639    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7657-2ff7df"]
Oct 30 21:19:41.760797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:41.760719    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-2ff7df"
Oct 30 21:19:41.764985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:41.764958    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:19:41.766100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:41.766073    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:19:41.766251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:41.766088    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9lmpp\""
Oct 30 21:19:41.778614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:41.778588    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn58t\" (UniqueName: \"kubernetes.io/projected/c24e34bb-272a-452e-a037-202f82c28a92-kube-api-access-qn58t\") pod \"test-curl-7657-2ff7df\" (UID: \"c24e34bb-272a-452e-a037-202f82c28a92\") " pod="opendatahub/test-curl-7657-2ff7df"
Oct 30 21:19:41.879695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:41.879667    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn58t\" (UniqueName: \"kubernetes.io/projected/c24e34bb-272a-452e-a037-202f82c28a92-kube-api-access-qn58t\") pod \"test-curl-7657-2ff7df\" (UID: \"c24e34bb-272a-452e-a037-202f82c28a92\") " pod="opendatahub/test-curl-7657-2ff7df"
Oct 30 21:19:41.888690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:41.888667    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn58t\" (UniqueName: \"kubernetes.io/projected/c24e34bb-272a-452e-a037-202f82c28a92-kube-api-access-qn58t\") pod \"test-curl-7657-2ff7df\" (UID: \"c24e34bb-272a-452e-a037-202f82c28a92\") " pod="opendatahub/test-curl-7657-2ff7df"
Oct 30 21:19:42.078605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.078510    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-2ff7df"
Oct 30 21:19:42.330597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.330491    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l" podUID="20a82ca5-2768-4089-a95e-453351c37254" containerName="payload-pre-processing" containerID="cri-o://05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7" gracePeriod=25
Oct 30 21:19:42.364706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.364676    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7657-2ff7df"]
Oct 30 21:19:42.387325 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:19:42.387299    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc24e34bb_272a_452e_a037_202f82c28a92.slice/crio-b4bf76b3ea585023a2d53d7c1f3b66dfc47e6678ee1376ec665aae32e938f5fc WatchSource:0}: Error finding container b4bf76b3ea585023a2d53d7c1f3b66dfc47e6678ee1376ec665aae32e938f5fc: Status 404 returned error can't find the container with id b4bf76b3ea585023a2d53d7c1f3b66dfc47e6678ee1376ec665aae32e938f5fc
Oct 30 21:19:42.437251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.437235    4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:19:42.593187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.593112    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"
Oct 30 21:19:42.603277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.603257    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"
Oct 30 21:19:42.619091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.619066    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a82ca5-2768-4089-a95e-453351c37254-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a82ca5-2768-4089-a95e-453351c37254-plugins-config-volume\") pod \"20a82ca5-2768-4089-a95e-453351c37254\" (UID: \"20a82ca5-2768-4089-a95e-453351c37254\") "
Oct 30 21:19:42.619225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.619212    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a82ca5-2768-4089-a95e-453351c37254-kube-api-access-p7wvr\" (UniqueName: \"kubernetes.io/projected/20a82ca5-2768-4089-a95e-453351c37254-kube-api-access-p7wvr\") pod \"20a82ca5-2768-4089-a95e-453351c37254\" (UID: \"20a82ca5-2768-4089-a95e-453351c37254\") "
Oct 30 21:19:42.619417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.619368    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a82ca5-2768-4089-a95e-453351c37254-plugins-config-volume" pod "20a82ca5-2768-4089-a95e-453351c37254" (UID: "20a82ca5-2768-4089-a95e-453351c37254"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:42.619875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.619859    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20a82ca5-2768-4089-a95e-453351c37254-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:42.621461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.621385    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a82ca5-2768-4089-a95e-453351c37254-kube-api-access-p7wvr" pod "20a82ca5-2768-4089-a95e-453351c37254" (UID: "20a82ca5-2768-4089-a95e-453351c37254"). InnerVolumeSpecName "kube-api-access-p7wvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:42.721311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.721275    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7wvr\" (UniqueName: \"kubernetes.io/projected/20a82ca5-2768-4089-a95e-453351c37254-kube-api-access-p7wvr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:42.849849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.849677    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7657-2ff7df" event={"ID":"c24e34bb-272a-452e-a037-202f82c28a92","Type":"ContainerStarted","Data":"b4bf76b3ea585023a2d53d7c1f3b66dfc47e6678ee1376ec665aae32e938f5fc"}
Oct 30 21:19:42.851114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.851091    4918 generic.go:356] "Generic (PLEG): container finished" podID="20a82ca5-2768-4089-a95e-453351c37254" containerID="05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7" exitCode=0
Oct 30 21:19:42.851215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.851132    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"
Oct 30 21:19:42.851215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.851180    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l" event={"ID":"20a82ca5-2768-4089-a95e-453351c37254","Type":"ContainerDied","Data":"05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7"}
Oct 30 21:19:42.851338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.851235    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l" event={"ID":"20a82ca5-2768-4089-a95e-453351c37254","Type":"ContainerDied","Data":"8306383620bc124e6efa9c6b252eb3157aa56e0bb9e4c8f3a1ca854bcc45be68"}
Oct 30 21:19:42.851338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.851265    4918 scope.go:122] "RemoveContainer" containerID="05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7"
Oct 30 21:19:42.862569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.862552    4918 scope.go:122] "RemoveContainer" containerID="05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7"
Oct 30 21:19:42.862875 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:19:42.862856    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7\": container with ID starting with 05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7 not found: ID does not exist" containerID="05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7"
Oct 30 21:19:42.862948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.862882    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7"} err="failed to get container status \"05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7\": rpc error: code = NotFound desc = could not find container \"05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7\": container with ID starting with 05ceac79c25e019261dc5477607837a4d308e4d855acb720f5ccc07fef66a4f7 not found: ID does not exist"
Oct 30 21:19:42.876583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.876562    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"]
Oct 30 21:19:42.883374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:42.883354    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c4aac-6d5985746c-lf82l"]
Oct 30 21:19:43.079737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:43.079707    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a82ca5-2768-4089-a95e-453351c37254" path="/var/lib/kubelet/pods/20a82ca5-2768-4089-a95e-453351c37254/volumes"
Oct 30 21:19:45.485491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:45.481842    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"]
Oct 30 21:19:46.947332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:46.947300    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7657-2ff7df" event={"ID":"c24e34bb-272a-452e-a037-202f82c28a92","Type":"ContainerStarted","Data":"9d28179cd368097cdd935c62b03b71de265257745cc9f5f185b3c66ac36f8502"}
Oct 30 21:19:46.969340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:46.969290    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7657-2ff7df" podStartSLOduration=2.388174547 podStartE2EDuration="5.96927517s" totalImagesPullingTime="3.581100623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:46.966045709 +0000 UTC m=+1100.012604872" watchObservedRunningTime="2026-10-30 21:19:46.96927517 +0000 UTC m=+1100.015834315"
Oct 30 21:19:47.747831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.747789    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7657-2ff7df"]
Oct 30 21:19:47.756977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.756948    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7657-2ff7df"]
Oct 30 21:19:47.961154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.961123    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"]
Oct 30 21:19:47.961759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.961745    4918 cpu_manager.go:414] "containerMap: removing container" podUID="20a82ca5-2768-4089-a95e-453351c37254" containerName="payload-pre-processing"
Oct 30 21:19:47.961803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.961759    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a82ca5-2768-4089-a95e-453351c37254" containerName="payload-pre-processing"
Oct 30 21:19:47.961846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.961810    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c24e34bb-272a-452e-a037-202f82c28a92" containerName="test-curl-7657-2ff7df"
Oct 30 21:19:47.961846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.961815    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24e34bb-272a-452e-a037-202f82c28a92" containerName="test-curl-7657-2ff7df"
Oct 30 21:19:47.961923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.961913    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24e34bb-272a-452e-a037-202f82c28a92" containerName="test-curl-7657-2ff7df"
Oct 30 21:19:47.961966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.961926    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a82ca5-2768-4089-a95e-453351c37254" containerName="payload-pre-processing"
Oct 30 21:19:47.965818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.965800    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:47.971421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.971369    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:19:47.971651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.971633    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:19:47.971737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.971712    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:19:47.971737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.971731    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:19:47.972979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:47.972957    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"]
Oct 30 21:19:48.119686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.119602    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfr92\" (UniqueName: \"kubernetes.io/projected/e7868065-ea4b-491a-89ef-8c7ced1d1a68-kube-api-access-gfr92\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl\" (UID: \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:48.119686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.119673    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl\" (UID: \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:48.119879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.119752    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl\" (UID: \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:48.221143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.221104    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfr92\" (UniqueName: \"kubernetes.io/projected/e7868065-ea4b-491a-89ef-8c7ced1d1a68-kube-api-access-gfr92\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl\" (UID: \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:48.221504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.221484    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl\" (UID: \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:48.221632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.221618    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl\" (UID: \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:48.222954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.222927    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl\" (UID: \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:48.227349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.227319    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl\" (UID: \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:48.231708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.231688    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfr92\" (UniqueName: \"kubernetes.io/projected/e7868065-ea4b-491a-89ef-8c7ced1d1a68-kube-api-access-gfr92\") pod \"maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl\" (UID: \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:48.317741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.317713    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:48.608424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.607985    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"]
Oct 30 21:19:48.952209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.952107    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"]
Oct 30 21:19:48.975952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.975918    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl" event={"ID":"e7868065-ea4b-491a-89ef-8c7ced1d1a68","Type":"ContainerStarted","Data":"8f136b9597e45af395c9704cf93430878003d55bc2d4efcb740ee7c13cab218c"}
Oct 30 21:19:48.976472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:48.976137    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7657-2ff7df" podUID="c24e34bb-272a-452e-a037-202f82c28a92" containerName="test-curl-7657-2ff7df" containerID="cri-o://9d28179cd368097cdd935c62b03b71de265257745cc9f5f185b3c66ac36f8502" gracePeriod=2
Oct 30 21:19:48.980429 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:19:48.980340    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7657-2ff7df\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="c24e34bb-272a-452e-a037-202f82c28a92" pod="opendatahub/test-curl-7657-2ff7df"
Oct 30 21:19:49.078701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.078671    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"]
Oct 30 21:19:49.084279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.084258    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"
Oct 30 21:19:49.115414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.114966    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4c6998-dockercfg-hlcf4\""
Oct 30 21:19:49.125856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.125811    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"]
Oct 30 21:19:49.162216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.162183    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e680b3-0724-436c-a33f-217385db1bd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz\" (UID: \"73e680b3-0724-436c-a33f-217385db1bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"
Oct 30 21:19:49.162376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.162338    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s57wp\" (UniqueName: \"kubernetes.io/projected/73e680b3-0724-436c-a33f-217385db1bd1-kube-api-access-s57wp\") pod \"payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz\" (UID: \"73e680b3-0724-436c-a33f-217385db1bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"
Oct 30 21:19:49.175834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.175802    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"]
Oct 30 21:19:49.183832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.183807    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"
Oct 30 21:19:49.231487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.231459    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"]
Oct 30 21:19:49.264839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.264813    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s57wp\" (UniqueName: \"kubernetes.io/projected/73e680b3-0724-436c-a33f-217385db1bd1-kube-api-access-s57wp\") pod \"payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz\" (UID: \"73e680b3-0724-436c-a33f-217385db1bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"
Oct 30 21:19:49.264962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.264898    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6012c0fa-471a-41ec-ba1a-95be1768af03-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9\" (UID: \"6012c0fa-471a-41ec-ba1a-95be1768af03\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"
Oct 30 21:19:49.265037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.265015    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52f6x\" (UniqueName: \"kubernetes.io/projected/6012c0fa-471a-41ec-ba1a-95be1768af03-kube-api-access-52f6x\") pod \"payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9\" (UID: \"6012c0fa-471a-41ec-ba1a-95be1768af03\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"
Oct 30 21:19:49.265161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.265148    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e680b3-0724-436c-a33f-217385db1bd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz\" (UID: \"73e680b3-0724-436c-a33f-217385db1bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"
Oct 30 21:19:49.265790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.265773    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e680b3-0724-436c-a33f-217385db1bd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz\" (UID: \"73e680b3-0724-436c-a33f-217385db1bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"
Oct 30 21:19:49.273274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.273254    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s57wp\" (UniqueName: \"kubernetes.io/projected/73e680b3-0724-436c-a33f-217385db1bd1-kube-api-access-s57wp\") pod \"payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz\" (UID: \"73e680b3-0724-436c-a33f-217385db1bd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"
Oct 30 21:19:49.381418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.368540    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6012c0fa-471a-41ec-ba1a-95be1768af03-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9\" (UID: \"6012c0fa-471a-41ec-ba1a-95be1768af03\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"
Oct 30 21:19:49.381418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.368664    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52f6x\" (UniqueName: \"kubernetes.io/projected/6012c0fa-471a-41ec-ba1a-95be1768af03-kube-api-access-52f6x\") pod \"payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9\" (UID: \"6012c0fa-471a-41ec-ba1a-95be1768af03\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"
Oct 30 21:19:49.395139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.387043    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6012c0fa-471a-41ec-ba1a-95be1768af03-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9\" (UID: \"6012c0fa-471a-41ec-ba1a-95be1768af03\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"
Oct 30 21:19:49.414414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.402076    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52f6x\" (UniqueName: \"kubernetes.io/projected/6012c0fa-471a-41ec-ba1a-95be1768af03-kube-api-access-52f6x\") pod \"payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9\" (UID: \"6012c0fa-471a-41ec-ba1a-95be1768af03\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"
Oct 30 21:19:49.451418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.434666    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"
Oct 30 21:19:49.615776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:49.615745    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"
Oct 30 21:19:50.016660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:50.016625    4918 generic.go:356] "Generic (PLEG): container finished" podID="e7868065-ea4b-491a-89ef-8c7ced1d1a68" containerID="80d5602e511e90fbbb9d2ffd6b399a9cbc45b8f3243c2ebe2f37d9db4441e279" exitCode=0
Oct 30 21:19:50.017513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:50.017113    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl" event={"ID":"e7868065-ea4b-491a-89ef-8c7ced1d1a68","Type":"ContainerDied","Data":"80d5602e511e90fbbb9d2ffd6b399a9cbc45b8f3243c2ebe2f37d9db4441e279"}
Oct 30 21:19:50.039121 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:19:50.039075    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e680b3_0724_436c_a33f_217385db1bd1.slice/crio-aa3a47b7b7871d92b67fee363ccbdbae9a5bb8105b2aa2b9fde0cc7b859517b9 WatchSource:0}: Error finding container aa3a47b7b7871d92b67fee363ccbdbae9a5bb8105b2aa2b9fde0cc7b859517b9: Status 404 returned error can't find the container with id aa3a47b7b7871d92b67fee363ccbdbae9a5bb8105b2aa2b9fde0cc7b859517b9
Oct 30 21:19:50.043282 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:50.043260    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"]
Oct 30 21:19:50.115365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:50.115341    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"]
Oct 30 21:19:50.116928 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:19:50.116906    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6012c0fa_471a_41ec_ba1a_95be1768af03.slice/crio-08577f4579dc9e1f5f374b0e5c4e65ad7e80004a4d325b3b02997cf8fdce2150 WatchSource:0}: Error finding container 08577f4579dc9e1f5f374b0e5c4e65ad7e80004a4d325b3b02997cf8fdce2150: Status 404 returned error can't find the container with id 08577f4579dc9e1f5f374b0e5c4e65ad7e80004a4d325b3b02997cf8fdce2150
Oct 30 21:19:50.482264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:50.482220    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz" podUID="533db7d6-d52a-4bec-a02d-9fab5ca8db9b" containerName="payload-processing" containerID="cri-o://98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f" gracePeriod=25
Oct 30 21:19:50.878584 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:50.878559    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"
Oct 30 21:19:50.931726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:50.931687    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-plugins-config-volume\") pod \"533db7d6-d52a-4bec-a02d-9fab5ca8db9b\" (UID: \"533db7d6-d52a-4bec-a02d-9fab5ca8db9b\") "
Oct 30 21:19:50.932382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:50.932264    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-plugins-config-volume" pod "533db7d6-d52a-4bec-a02d-9fab5ca8db9b" (UID: "533db7d6-d52a-4bec-a02d-9fab5ca8db9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:50.936883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:50.935310    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-kube-api-access-jmj5c\" (UniqueName: \"kubernetes.io/projected/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-kube-api-access-jmj5c\") pod \"533db7d6-d52a-4bec-a02d-9fab5ca8db9b\" (UID: \"533db7d6-d52a-4bec-a02d-9fab5ca8db9b\") "
Oct 30 21:19:50.936883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:50.936033    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:50.938819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:50.938792    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-kube-api-access-jmj5c" pod "533db7d6-d52a-4bec-a02d-9fab5ca8db9b" (UID: "533db7d6-d52a-4bec-a02d-9fab5ca8db9b"). InnerVolumeSpecName "kube-api-access-jmj5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:51.139284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.139189    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmj5c\" (UniqueName: \"kubernetes.io/projected/533db7d6-d52a-4bec-a02d-9fab5ca8db9b-kube-api-access-jmj5c\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:51.179437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.175783    4918 generic.go:356] "Generic (PLEG): container finished" podID="533db7d6-d52a-4bec-a02d-9fab5ca8db9b" containerID="98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f" exitCode=0
Oct 30 21:19:51.179437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.176045    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz" event={"ID":"533db7d6-d52a-4bec-a02d-9fab5ca8db9b","Type":"ContainerDied","Data":"98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f"}
Oct 30 21:19:51.179437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.176087    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz" event={"ID":"533db7d6-d52a-4bec-a02d-9fab5ca8db9b","Type":"ContainerDied","Data":"1b1122f73814f2ed9a9dd6093cbcdb78bed6bd61b45d3d5b25a57fafdec8b935"}
Oct 30 21:19:51.179437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.176117    4918 scope.go:122] "RemoveContainer" containerID="98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f"
Oct 30 21:19:51.179437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.176263    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"
Oct 30 21:19:51.236678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.231607    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9" event={"ID":"6012c0fa-471a-41ec-ba1a-95be1768af03","Type":"ContainerStarted","Data":"08577f4579dc9e1f5f374b0e5c4e65ad7e80004a4d325b3b02997cf8fdce2150"}
Oct 30 21:19:51.244790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.244761    4918 generic.go:356] "Generic (PLEG): container finished" podID="c24e34bb-272a-452e-a037-202f82c28a92" containerID="9d28179cd368097cdd935c62b03b71de265257745cc9f5f185b3c66ac36f8502" exitCode=137
Oct 30 21:19:51.250489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.250459    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz" event={"ID":"73e680b3-0724-436c-a33f-217385db1bd1","Type":"ContainerStarted","Data":"aa3a47b7b7871d92b67fee363ccbdbae9a5bb8105b2aa2b9fde0cc7b859517b9"}
Oct 30 21:19:51.261358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.261271    4918 scope.go:122] "RemoveContainer" containerID="98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f"
Oct 30 21:19:51.262166 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:19:51.262091    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f\": container with ID starting with 98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f not found: ID does not exist" containerID="98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f"
Oct 30 21:19:51.262166 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.262132    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f"} err="failed to get container status \"98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f\": rpc error: code = NotFound desc = could not find container \"98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f\": container with ID starting with 98a66f72ccbe95f3500c302189f6330857e4123564b217571117015e5d4af08f not found: ID does not exist"
Oct 30 21:19:51.274523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.274458    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"]
Oct 30 21:19:51.285264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.285235    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c6998-54455885dc-xrdrz"]
Oct 30 21:19:51.477266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.477235    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-2ff7df"
Oct 30 21:19:51.547271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.547243    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24e34bb-272a-452e-a037-202f82c28a92-kube-api-access-qn58t\" (UniqueName: \"kubernetes.io/projected/c24e34bb-272a-452e-a037-202f82c28a92-kube-api-access-qn58t\") pod \"c24e34bb-272a-452e-a037-202f82c28a92\" (UID: \"c24e34bb-272a-452e-a037-202f82c28a92\") "
Oct 30 21:19:51.551955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.551925    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24e34bb-272a-452e-a037-202f82c28a92-kube-api-access-qn58t" pod "c24e34bb-272a-452e-a037-202f82c28a92" (UID: "c24e34bb-272a-452e-a037-202f82c28a92"). InnerVolumeSpecName "kube-api-access-qn58t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:51.568661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.568633    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:51.648765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.648168    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-openshift-ca\") pod \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\" (UID: \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\") "
Oct 30 21:19:51.648765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.648355    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7868065-ea4b-491a-89ef-8c7ced1d1a68-kube-api-access-gfr92\" (UniqueName: \"kubernetes.io/projected/e7868065-ea4b-491a-89ef-8c7ced1d1a68-kube-api-access-gfr92\") pod \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\" (UID: \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\") "
Oct 30 21:19:51.648765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.648413    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-xks-ca\") pod \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\" (UID: \"e7868065-ea4b-491a-89ef-8c7ced1d1a68\") "
Oct 30 21:19:51.648765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.648511    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-openshift-ca" pod "e7868065-ea4b-491a-89ef-8c7ced1d1a68" (UID: "e7868065-ea4b-491a-89ef-8c7ced1d1a68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:51.649293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.649196    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:51.649293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.649214    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn58t\" (UniqueName: \"kubernetes.io/projected/c24e34bb-272a-452e-a037-202f82c28a92-kube-api-access-qn58t\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:51.651639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.651612    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-xks-ca" pod "e7868065-ea4b-491a-89ef-8c7ced1d1a68" (UID: "e7868065-ea4b-491a-89ef-8c7ced1d1a68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:19:51.652306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.652281    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7868065-ea4b-491a-89ef-8c7ced1d1a68-kube-api-access-gfr92" pod "e7868065-ea4b-491a-89ef-8c7ced1d1a68" (UID: "e7868065-ea4b-491a-89ef-8c7ced1d1a68"). InnerVolumeSpecName "kube-api-access-gfr92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:51.768416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.761355    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfr92\" (UniqueName: \"kubernetes.io/projected/e7868065-ea4b-491a-89ef-8c7ced1d1a68-kube-api-access-gfr92\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:51.768416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:51.761406    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7868065-ea4b-491a-89ef-8c7ced1d1a68-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:52.276558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:52.276529    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-2ff7df"
Oct 30 21:19:52.277038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:52.276569    4918 scope.go:122] "RemoveContainer" containerID="9d28179cd368097cdd935c62b03b71de265257745cc9f5f185b3c66ac36f8502"
Oct 30 21:19:52.279328 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:52.279296    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl" event={"ID":"e7868065-ea4b-491a-89ef-8c7ced1d1a68","Type":"ContainerDied","Data":"8f136b9597e45af395c9704cf93430878003d55bc2d4efcb740ee7c13cab218c"}
Oct 30 21:19:52.279474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:52.279353    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f136b9597e45af395c9704cf93430878003d55bc2d4efcb740ee7c13cab218c"
Oct 30 21:19:52.279530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:52.279493    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"
Oct 30 21:19:52.295802 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:19:52.295650    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7657-2ff7df\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="c24e34bb-272a-452e-a037-202f82c28a92" pod="opendatahub/test-curl-7657-2ff7df"
Oct 30 21:19:52.304812 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:19:52.304696    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7657-2ff7df\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="c24e34bb-272a-452e-a037-202f82c28a92" pod="opendatahub/test-curl-7657-2ff7df"
Oct 30 21:19:53.140492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:53.135596    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533db7d6-d52a-4bec-a02d-9fab5ca8db9b" path="/var/lib/kubelet/pods/533db7d6-d52a-4bec-a02d-9fab5ca8db9b/volumes"
Oct 30 21:19:53.140492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:53.137474    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24e34bb-272a-452e-a037-202f82c28a92" path="/var/lib/kubelet/pods/c24e34bb-272a-452e-a037-202f82c28a92/volumes"
Oct 30 21:19:53.952712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:53.952667    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh" podUID="513ace8b-991a-4516-a054-20e8b568f6e3" containerName="payload-pre-processing" containerID="cri-o://bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388" gracePeriod=25
Oct 30 21:19:54.320525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.320491    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"]
Oct 30 21:19:54.321511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.321488    4918 cpu_manager.go:414] "containerMap: removing container" podUID="533db7d6-d52a-4bec-a02d-9fab5ca8db9b" containerName="payload-processing"
Oct 30 21:19:54.321511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.321511    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533db7d6-d52a-4bec-a02d-9fab5ca8db9b" containerName="payload-processing"
Oct 30 21:19:54.321710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.321545    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e7868065-ea4b-491a-89ef-8c7ced1d1a68" containerName="revoke-keys"
Oct 30 21:19:54.321710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.321553    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7868065-ea4b-491a-89ef-8c7ced1d1a68" containerName="revoke-keys"
Oct 30 21:19:54.321814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.321751    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="533db7d6-d52a-4bec-a02d-9fab5ca8db9b" containerName="payload-processing"
Oct 30 21:19:54.321814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.321766    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7868065-ea4b-491a-89ef-8c7ced1d1a68" containerName="revoke-keys"
Oct 30 21:19:54.326810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.326787    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"
Oct 30 21:19:54.327127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.327112    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"]
Oct 30 21:19:54.339414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.336335    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"]
Oct 30 21:19:54.339414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.338779    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"
Oct 30 21:19:54.343580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.343557    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"]
Oct 30 21:19:54.343713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.343636    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"
Oct 30 21:19:54.347511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.347482    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9" event={"ID":"6012c0fa-471a-41ec-ba1a-95be1768af03","Type":"ContainerStarted","Data":"e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91"}
Oct 30 21:19:54.347786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.347767    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"
Oct 30 21:19:54.348512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.348476    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3ac6bf-dockercfg-d6dq7\""
Oct 30 21:19:54.352573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.352020    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zhc7\" (UniqueName: \"kubernetes.io/projected/edebd91a-ce2a-4544-afa3-f1828eb16f8c-kube-api-access-5zhc7\") pod \"payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk\" (UID: \"edebd91a-ce2a-4544-afa3-f1828eb16f8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"
Oct 30 21:19:54.352573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.352184    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edebd91a-ce2a-4544-afa3-f1828eb16f8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk\" (UID: \"edebd91a-ce2a-4544-afa3-f1828eb16f8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"
Oct 30 21:19:54.353760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.353656    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz" event={"ID":"73e680b3-0724-436c-a33f-217385db1bd1","Type":"ContainerStarted","Data":"089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f"}
Oct 30 21:19:54.353868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.353772    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"
Oct 30 21:19:54.356849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.356741    4918 generic.go:356] "Generic (PLEG): container finished" podID="513ace8b-991a-4516-a054-20e8b568f6e3" containerID="bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388" exitCode=0
Oct 30 21:19:54.356849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.356816    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh" event={"ID":"513ace8b-991a-4516-a054-20e8b568f6e3","Type":"ContainerDied","Data":"bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388"}
Oct 30 21:19:54.357162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.357100    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh" event={"ID":"513ace8b-991a-4516-a054-20e8b568f6e3","Type":"ContainerDied","Data":"4756460a513b3b7f7be7dfa7dcb5307d949701f15617e4ffc3884d7f6670ecb5"}
Oct 30 21:19:54.357433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.357333    4918 scope.go:122] "RemoveContainer" containerID="bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388"
Oct 30 21:19:54.357759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.357695    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"
Oct 30 21:19:54.392475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.392451    4918 scope.go:122] "RemoveContainer" containerID="bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388"
Oct 30 21:19:54.392867 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:19:54.392829    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388\": container with ID starting with bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388 not found: ID does not exist" containerID="bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388"
Oct 30 21:19:54.393037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.392866    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388"} err="failed to get container status \"bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388\": rpc error: code = NotFound desc = could not find container \"bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388\": container with ID starting with bc112d6e95c330ba3c438ade6737b3eb6e3459c53a0230a2914cb19d45288388 not found: ID does not exist"
Oct 30 21:19:54.397002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.396950    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz" podStartSLOduration=2.149919725 podStartE2EDuration="5.396933844s" totalImagesPullingTime="3.247014119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:54.393061396 +0000 UTC m=+1107.439620544" watchObservedRunningTime="2026-10-30 21:19:54.396933844 +0000 UTC m=+1107.443492989"
Oct 30 21:19:54.434270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.434203    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9" podStartSLOduration=2.226495654 podStartE2EDuration="5.434183795s" totalImagesPullingTime="3.207688141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:54.424299546 +0000 UTC m=+1107.470858701" watchObservedRunningTime="2026-10-30 21:19:54.434183795 +0000 UTC m=+1107.480742941"
Oct 30 21:19:54.454379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.453332    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/513ace8b-991a-4516-a054-20e8b568f6e3-kube-api-access-bg86k\" (UniqueName: \"kubernetes.io/projected/513ace8b-991a-4516-a054-20e8b568f6e3-kube-api-access-bg86k\") pod \"513ace8b-991a-4516-a054-20e8b568f6e3\" (UID: \"513ace8b-991a-4516-a054-20e8b568f6e3\") "
Oct 30 21:19:54.454379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.453458    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/513ace8b-991a-4516-a054-20e8b568f6e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513ace8b-991a-4516-a054-20e8b568f6e3-plugins-config-volume\") pod \"513ace8b-991a-4516-a054-20e8b568f6e3\" (UID: \"513ace8b-991a-4516-a054-20e8b568f6e3\") "
Oct 30 21:19:54.454379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.453718    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zhc7\" (UniqueName: \"kubernetes.io/projected/edebd91a-ce2a-4544-afa3-f1828eb16f8c-kube-api-access-5zhc7\") pod \"payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk\" (UID: \"edebd91a-ce2a-4544-afa3-f1828eb16f8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"
Oct 30 21:19:54.454379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.453883    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edebd91a-ce2a-4544-afa3-f1828eb16f8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk\" (UID: \"edebd91a-ce2a-4544-afa3-f1828eb16f8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"
Oct 30 21:19:54.454379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.453963    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hppb\" (UniqueName: \"kubernetes.io/projected/4405372a-da79-4d92-95a9-9261167ce3dd-kube-api-access-7hppb\") pod \"payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62\" (UID: \"4405372a-da79-4d92-95a9-9261167ce3dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"
Oct 30 21:19:54.454379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.454059    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513ace8b-991a-4516-a054-20e8b568f6e3-plugins-config-volume" pod "513ace8b-991a-4516-a054-20e8b568f6e3" (UID: "513ace8b-991a-4516-a054-20e8b568f6e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:19:54.454864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.454563    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4405372a-da79-4d92-95a9-9261167ce3dd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62\" (UID: \"4405372a-da79-4d92-95a9-9261167ce3dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"
Oct 30 21:19:54.455099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.454953    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/513ace8b-991a-4516-a054-20e8b568f6e3-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:54.455099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.455069    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edebd91a-ce2a-4544-afa3-f1828eb16f8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk\" (UID: \"edebd91a-ce2a-4544-afa3-f1828eb16f8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"
Oct 30 21:19:54.458572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.458541    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513ace8b-991a-4516-a054-20e8b568f6e3-kube-api-access-bg86k" pod "513ace8b-991a-4516-a054-20e8b568f6e3" (UID: "513ace8b-991a-4516-a054-20e8b568f6e3"). InnerVolumeSpecName "kube-api-access-bg86k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:19:54.470153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.470124    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zhc7\" (UniqueName: \"kubernetes.io/projected/edebd91a-ce2a-4544-afa3-f1828eb16f8c-kube-api-access-5zhc7\") pod \"payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk\" (UID: \"edebd91a-ce2a-4544-afa3-f1828eb16f8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"
Oct 30 21:19:54.560607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.558857    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4405372a-da79-4d92-95a9-9261167ce3dd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62\" (UID: \"4405372a-da79-4d92-95a9-9261167ce3dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"
Oct 30 21:19:54.560607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.559141    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hppb\" (UniqueName: \"kubernetes.io/projected/4405372a-da79-4d92-95a9-9261167ce3dd-kube-api-access-7hppb\") pod \"payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62\" (UID: \"4405372a-da79-4d92-95a9-9261167ce3dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"
Oct 30 21:19:54.560607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.559332    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg86k\" (UniqueName: \"kubernetes.io/projected/513ace8b-991a-4516-a054-20e8b568f6e3-kube-api-access-bg86k\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:19:54.573201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.568465    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4405372a-da79-4d92-95a9-9261167ce3dd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62\" (UID: \"4405372a-da79-4d92-95a9-9261167ce3dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"
Oct 30 21:19:54.656211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.656178    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hppb\" (UniqueName: \"kubernetes.io/projected/4405372a-da79-4d92-95a9-9261167ce3dd-kube-api-access-7hppb\") pod \"payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62\" (UID: \"4405372a-da79-4d92-95a9-9261167ce3dd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"
Oct 30 21:19:54.662583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.656641    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"
Oct 30 21:19:54.666763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:54.666739    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"
Oct 30 21:19:55.079796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:55.079726    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"]
Oct 30 21:19:55.079796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:55.079765    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"]
Oct 30 21:19:55.086148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:55.086121    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0c4aac-ktj5w"]
Oct 30 21:19:55.090751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:55.090723    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0fbd348-66v4s"]
Oct 30 21:19:55.095221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:55.095153    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"]
Oct 30 21:19:55.099593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:55.099286    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-7c7ff4cf-dmrrh"]
Oct 30 21:19:55.352289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:55.352133    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"]
Oct 30 21:19:55.360829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:55.360799    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"]
Oct 30 21:19:56.398084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:56.398046    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62" event={"ID":"4405372a-da79-4d92-95a9-9261167ce3dd","Type":"ContainerStarted","Data":"32ce61271d1b41dd8175fdb9a4186c6cbf5cb98d22e28818dec0ae83495fdcbf"}
Oct 30 21:19:56.398692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:56.398118    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62" event={"ID":"4405372a-da79-4d92-95a9-9261167ce3dd","Type":"ContainerStarted","Data":"20d1642b369bcab3a5a4a86ec6985d424384002ab72962071ca90bdaf6fc53b0"}
Oct 30 21:19:56.398692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:56.398153    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"
Oct 30 21:19:56.400130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:56.400100    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk" event={"ID":"edebd91a-ce2a-4544-afa3-f1828eb16f8c","Type":"ContainerStarted","Data":"ff779a4b63502dde12933ebd1aabcbf44df5df6c57c9958a2c1bdda3a3ee315b"}
Oct 30 21:19:56.400266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:56.400153    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk" event={"ID":"edebd91a-ce2a-4544-afa3-f1828eb16f8c","Type":"ContainerStarted","Data":"594bc786f9e1f06a92d8bb17272fe53e36f6f849dd565672116f06249f51fde7"}
Oct 30 21:19:56.400266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:56.400183    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"
Oct 30 21:19:56.422528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:56.422484    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62" podStartSLOduration=2.422470745 podStartE2EDuration="2.422470745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:56.415853334 +0000 UTC m=+1109.462412504" watchObservedRunningTime="2026-10-30 21:19:56.422470745 +0000 UTC m=+1109.469029889"
Oct 30 21:19:56.436525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:56.436485    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk" podStartSLOduration=2.436475514 podStartE2EDuration="2.436475514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:19:56.433972799 +0000 UTC m=+1109.480531971" watchObservedRunningTime="2026-10-30 21:19:56.436475514 +0000 UTC m=+1109.483034659"
Oct 30 21:19:57.078726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:57.078692    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25659f1b-4552-4454-996f-58b149dfb0a7" path="/var/lib/kubelet/pods/25659f1b-4552-4454-996f-58b149dfb0a7/volumes"
Oct 30 21:19:57.079125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:57.079112    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="513ace8b-991a-4516-a054-20e8b568f6e3" path="/var/lib/kubelet/pods/513ace8b-991a-4516-a054-20e8b568f6e3/volumes"
Oct 30 21:19:57.079543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:57.079530    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5" path="/var/lib/kubelet/pods/9a45cecb-a9bc-4c06-a0ff-1f87f51b0ab5/volumes"
Oct 30 21:19:57.412546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:57.412457    4918 generic.go:356] "Generic (PLEG): container finished" podID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerID="dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa" exitCode=0
Oct 30 21:19:57.412980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:57.412531    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" event={"ID":"81ab1dc4-eb49-4815-8c11-00f9cf59602a","Type":"ContainerDied","Data":"dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa"}
Oct 30 21:19:57.413076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:57.413060    4918 scope.go:122] "RemoveContainer" containerID="dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa"
Oct 30 21:19:57.414721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:57.414694    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:19:57.414834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:57.414764    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:19:57.414834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:57.414796    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:19:58.135584 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:58.135547    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"]
Oct 30 21:19:58.154674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:58.154645    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"]
Oct 30 21:19:58.514169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:58.514131    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" event={"ID":"81ab1dc4-eb49-4815-8c11-00f9cf59602a","Type":"ContainerStarted","Data":"6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2"}
Oct 30 21:19:58.950668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:58.950566    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:19:58.951937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:58.951903    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:19:58.952054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:58.951949    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:19:59.185288 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:19:59.185234    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:19:59.957868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:59.957830    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:19:59.958509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:19:59.958472    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:20:00.964518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:00.964108    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:20:00.966016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:00.965978    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:20:01.488414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:01.473048    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"]
Oct 30 21:20:01.488414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:01.478578    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5bc8c60a-7jkpl"]
Oct 30 21:20:01.950897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:01.950809    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:20:01.950897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:01.950863    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:20:02.984786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:02.984750    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:20:02.986573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:02.986539    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:20:03.207284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.205750    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x" podUID="1f2c2d6b-cd9f-4c54-86d2-d897d3056800" containerName="payload-processing" containerID="cri-o://7cc0a00b3ce524316d7764f4027ec09c2417931a675951b5047c8d8b6607346d" gracePeriod=25
Oct 30 21:20:03.207284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.206182    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4" podUID="af040445-e395-46fd-9e3d-a617d0d6ad94" containerName="payload-pre-processing" containerID="cri-o://a9163864c8a5af653c57fa06d7cdd8d45221d6fc1f8d0c93ffbd3b0e8f765aa2" gracePeriod=25
Oct 30 21:20:03.276055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.274198    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7868065-ea4b-491a-89ef-8c7ced1d1a68" path="/var/lib/kubelet/pods/e7868065-ea4b-491a-89ef-8c7ced1d1a68/volumes"
Oct 30 21:20:03.384976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.383948    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"]
Oct 30 21:20:03.417427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.415041    4918 cpu_manager.go:414] "containerMap: removing container" podUID="513ace8b-991a-4516-a054-20e8b568f6e3" containerName="payload-pre-processing"
Oct 30 21:20:03.417427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.415073    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513ace8b-991a-4516-a054-20e8b568f6e3" containerName="payload-pre-processing"
Oct 30 21:20:03.417427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.415299    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="513ace8b-991a-4516-a054-20e8b568f6e3" containerName="payload-pre-processing"
Oct 30 21:20:03.449108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.439467    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:03.449559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.449454    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:20:03.449754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.449714    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:20:03.455509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.455476    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"]
Oct 30 21:20:03.515475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.501720    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:20:03.515475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.501795    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:20:03.528466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.517156    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7\" (UID: \"80944c45-978b-4926-b5ca-24eada1bc4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:03.528466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.517251    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7\" (UID: \"80944c45-978b-4926-b5ca-24eada1bc4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:03.528466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.517463    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55dc2\" (UniqueName: \"kubernetes.io/projected/80944c45-978b-4926-b5ca-24eada1bc4e9-kube-api-access-55dc2\") pod \"maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7\" (UID: \"80944c45-978b-4926-b5ca-24eada1bc4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:03.623497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.622296    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55dc2\" (UniqueName: \"kubernetes.io/projected/80944c45-978b-4926-b5ca-24eada1bc4e9-kube-api-access-55dc2\") pod \"maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7\" (UID: \"80944c45-978b-4926-b5ca-24eada1bc4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:03.623497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.622406    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7\" (UID: \"80944c45-978b-4926-b5ca-24eada1bc4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:03.623497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.622501    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7\" (UID: \"80944c45-978b-4926-b5ca-24eada1bc4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:03.647527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.646366    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7\" (UID: \"80944c45-978b-4926-b5ca-24eada1bc4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:03.648110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.648061    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7\" (UID: \"80944c45-978b-4926-b5ca-24eada1bc4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:03.652220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.652173    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55dc2\" (UniqueName: \"kubernetes.io/projected/80944c45-978b-4926-b5ca-24eada1bc4e9-kube-api-access-55dc2\") pod \"maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7\" (UID: \"80944c45-978b-4926-b5ca-24eada1bc4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:03.784687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.780868    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:03.954988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.954952    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:20:03.961597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.956895    4918 generic.go:356] "Generic (PLEG): container finished" podID="1f2c2d6b-cd9f-4c54-86d2-d897d3056800" containerID="7cc0a00b3ce524316d7764f4027ec09c2417931a675951b5047c8d8b6607346d" exitCode=0
Oct 30 21:20:03.961990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.956927    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x" event={"ID":"1f2c2d6b-cd9f-4c54-86d2-d897d3056800","Type":"ContainerDied","Data":"7cc0a00b3ce524316d7764f4027ec09c2417931a675951b5047c8d8b6607346d"}
Oct 30 21:20:03.962522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.962480    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:20:03.969088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.965741    4918 generic.go:356] "Generic (PLEG): container finished" podID="af040445-e395-46fd-9e3d-a617d0d6ad94" containerID="a9163864c8a5af653c57fa06d7cdd8d45221d6fc1f8d0c93ffbd3b0e8f765aa2" exitCode=0
Oct 30 21:20:03.969088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.965874    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4" event={"ID":"af040445-e395-46fd-9e3d-a617d0d6ad94","Type":"ContainerDied","Data":"a9163864c8a5af653c57fa06d7cdd8d45221d6fc1f8d0c93ffbd3b0e8f765aa2"}
Oct 30 21:20:03.969088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.967877    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"
Oct 30 21:20:03.995040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:03.994577    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"
Oct 30 21:20:04.044467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.007296    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-kube-api-access-79p4p\" (UniqueName: \"kubernetes.io/projected/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-kube-api-access-79p4p\") pod \"1f2c2d6b-cd9f-4c54-86d2-d897d3056800\" (UID: \"1f2c2d6b-cd9f-4c54-86d2-d897d3056800\") "
Oct 30 21:20:04.044467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.042638    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af040445-e395-46fd-9e3d-a617d0d6ad94-kube-api-access-z2xvf\" (UniqueName: \"kubernetes.io/projected/af040445-e395-46fd-9e3d-a617d0d6ad94-kube-api-access-z2xvf\") pod \"af040445-e395-46fd-9e3d-a617d0d6ad94\" (UID: \"af040445-e395-46fd-9e3d-a617d0d6ad94\") "
Oct 30 21:20:04.044467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.042822    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-plugins-config-volume\") pod \"1f2c2d6b-cd9f-4c54-86d2-d897d3056800\" (UID: \"1f2c2d6b-cd9f-4c54-86d2-d897d3056800\") "
Oct 30 21:20:04.044467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.042885    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af040445-e395-46fd-9e3d-a617d0d6ad94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af040445-e395-46fd-9e3d-a617d0d6ad94-plugins-config-volume\") pod \"af040445-e395-46fd-9e3d-a617d0d6ad94\" (UID: \"af040445-e395-46fd-9e3d-a617d0d6ad94\") "
Oct 30 21:20:04.044467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.043914    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af040445-e395-46fd-9e3d-a617d0d6ad94-plugins-config-volume" pod "af040445-e395-46fd-9e3d-a617d0d6ad94" (UID: "af040445-e395-46fd-9e3d-a617d0d6ad94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:04.045332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.045299    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-plugins-config-volume" pod "1f2c2d6b-cd9f-4c54-86d2-d897d3056800" (UID: "1f2c2d6b-cd9f-4c54-86d2-d897d3056800"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:04.045656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.045632    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:04.045766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.045662    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af040445-e395-46fd-9e3d-a617d0d6ad94-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:04.048140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.048111    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af040445-e395-46fd-9e3d-a617d0d6ad94-kube-api-access-z2xvf" pod "af040445-e395-46fd-9e3d-a617d0d6ad94" (UID: "af040445-e395-46fd-9e3d-a617d0d6ad94"). InnerVolumeSpecName "kube-api-access-z2xvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:04.048342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.048321    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-kube-api-access-79p4p" pod "1f2c2d6b-cd9f-4c54-86d2-d897d3056800" (UID: "1f2c2d6b-cd9f-4c54-86d2-d897d3056800"). InnerVolumeSpecName "kube-api-access-79p4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:04.149431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.148069    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79p4p\" (UniqueName: \"kubernetes.io/projected/1f2c2d6b-cd9f-4c54-86d2-d897d3056800-kube-api-access-79p4p\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:04.149431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.148101    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2xvf\" (UniqueName: \"kubernetes.io/projected/af040445-e395-46fd-9e3d-a617d0d6ad94-kube-api-access-z2xvf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:04.362212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.361505    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"]
Oct 30 21:20:04.362212 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:20:04.361515    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80944c45_978b_4926_b5ca_24eada1bc4e9.slice/crio-20c141131ec72af97ab1ed6a82cae843d46e979cd58615aacaebf4c44c59079b WatchSource:0}: Error finding container 20c141131ec72af97ab1ed6a82cae843d46e979cd58615aacaebf4c44c59079b: Status 404 returned error can't find the container with id 20c141131ec72af97ab1ed6a82cae843d46e979cd58615aacaebf4c44c59079b
Oct 30 21:20:04.942924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.942849    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:20:04.955179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.955155    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:20:04.974930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.974907    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"
Oct 30 21:20:04.975055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.974912    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x" event={"ID":"1f2c2d6b-cd9f-4c54-86d2-d897d3056800","Type":"ContainerDied","Data":"22017dec47dcd6c385d7ced81a861cf4493903d7fb2ba0551b20afa57d21d8fd"}
Oct 30 21:20:04.975055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.975043    4918 scope.go:122] "RemoveContainer" containerID="7cc0a00b3ce524316d7764f4027ec09c2417931a675951b5047c8d8b6607346d"
Oct 30 21:20:04.976507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.976478    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4" event={"ID":"af040445-e395-46fd-9e3d-a617d0d6ad94","Type":"ContainerDied","Data":"8df7152f528c04c95a26ef79840f344b89ccb02f27c7dd6334ba1e60d17ee68b"}
Oct 30 21:20:04.976626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.976482    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"
Oct 30 21:20:04.984628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.984602    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7" event={"ID":"80944c45-978b-4926-b5ca-24eada1bc4e9","Type":"ContainerStarted","Data":"053606bb4dc2244c12bfb187bb0de8f09affc5d384a57aad325e11494f199d8d"}
Oct 30 21:20:04.984787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:04.984662    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7" event={"ID":"80944c45-978b-4926-b5ca-24eada1bc4e9","Type":"ContainerStarted","Data":"20c141131ec72af97ab1ed6a82cae843d46e979cd58615aacaebf4c44c59079b"}
Oct 30 21:20:05.006787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.006765    4918 scope.go:122] "RemoveContainer" containerID="a9163864c8a5af653c57fa06d7cdd8d45221d6fc1f8d0c93ffbd3b0e8f765aa2"
Oct 30 21:20:05.007326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.006997    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:20:05.030595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.030565    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"]
Oct 30 21:20:05.035523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.035501    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5bc8c60a-7cf5c7f8c-qpjq4"]
Oct 30 21:20:05.046403 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.046372    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"]
Oct 30 21:20:05.050621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.050598    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5bc8c60a-6f5f8df787-vsx9x"]
Oct 30 21:20:05.084242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.084182    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7" podStartSLOduration=2.084163286 podStartE2EDuration="2.084163286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:05.080157066 +0000 UTC m=+1118.126716215" watchObservedRunningTime="2026-10-30 21:20:05.084163286 +0000 UTC m=+1118.130722433"
Oct 30 21:20:05.088495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.088465    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2c2d6b-cd9f-4c54-86d2-d897d3056800" path="/var/lib/kubelet/pods/1f2c2d6b-cd9f-4c54-86d2-d897d3056800/volumes"
Oct 30 21:20:05.089080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.089055    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af040445-e395-46fd-9e3d-a617d0d6ad94" path="/var/lib/kubelet/pods/af040445-e395-46fd-9e3d-a617d0d6ad94/volumes"
Oct 30 21:20:05.391203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.391171    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"
Oct 30 21:20:05.391671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.391649    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"
Oct 30 21:20:05.997133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.997104    4918 generic.go:356] "Generic (PLEG): container finished" podID="80944c45-978b-4926-b5ca-24eada1bc4e9" containerID="053606bb4dc2244c12bfb187bb0de8f09affc5d384a57aad325e11494f199d8d" exitCode=0
Oct 30 21:20:05.997294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:05.997198    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7" event={"ID":"80944c45-978b-4926-b5ca-24eada1bc4e9","Type":"ContainerDied","Data":"053606bb4dc2244c12bfb187bb0de8f09affc5d384a57aad325e11494f199d8d"}
Oct 30 21:20:07.028994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.028959    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"]
Oct 30 21:20:07.036779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.036747    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a48a8269-sgscm"]
Oct 30 21:20:07.082973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.082249    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9dbe6fd-1277-4644-9906-0fb0797bb199" path="/var/lib/kubelet/pods/d9dbe6fd-1277-4644-9906-0fb0797bb199/volumes"
Oct 30 21:20:07.186534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.186502    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:07.240115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.240084    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-openshift-ca\") pod \"80944c45-978b-4926-b5ca-24eada1bc4e9\" (UID: \"80944c45-978b-4926-b5ca-24eada1bc4e9\") "
Oct 30 21:20:07.240535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.240505    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80944c45-978b-4926-b5ca-24eada1bc4e9-kube-api-access-55dc2\" (UniqueName: \"kubernetes.io/projected/80944c45-978b-4926-b5ca-24eada1bc4e9-kube-api-access-55dc2\") pod \"80944c45-978b-4926-b5ca-24eada1bc4e9\" (UID: \"80944c45-978b-4926-b5ca-24eada1bc4e9\") "
Oct 30 21:20:07.240678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.240541    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-openshift-ca" pod "80944c45-978b-4926-b5ca-24eada1bc4e9" (UID: "80944c45-978b-4926-b5ca-24eada1bc4e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:07.240678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.240567    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-xks-ca\") pod \"80944c45-978b-4926-b5ca-24eada1bc4e9\" (UID: \"80944c45-978b-4926-b5ca-24eada1bc4e9\") "
Oct 30 21:20:07.241222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.241196    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:07.243282 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.243248    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-xks-ca" pod "80944c45-978b-4926-b5ca-24eada1bc4e9" (UID: "80944c45-978b-4926-b5ca-24eada1bc4e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:20:07.244381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.244343    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80944c45-978b-4926-b5ca-24eada1bc4e9-kube-api-access-55dc2" pod "80944c45-978b-4926-b5ca-24eada1bc4e9" (UID: "80944c45-978b-4926-b5ca-24eada1bc4e9"). InnerVolumeSpecName "kube-api-access-55dc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:07.343614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.343421    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55dc2\" (UniqueName: \"kubernetes.io/projected/80944c45-978b-4926-b5ca-24eada1bc4e9-kube-api-access-55dc2\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:07.343614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.343458    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80944c45-978b-4926-b5ca-24eada1bc4e9-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:07.414753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.414721    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"
Oct 30 21:20:07.414967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:07.414806    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"
Oct 30 21:20:08.227448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:08.213221    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7" event={"ID":"80944c45-978b-4926-b5ca-24eada1bc4e9","Type":"ContainerDied","Data":"20c141131ec72af97ab1ed6a82cae843d46e979cd58615aacaebf4c44c59079b"}
Oct 30 21:20:08.227448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:08.213284    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20c141131ec72af97ab1ed6a82cae843d46e979cd58615aacaebf4c44c59079b"
Oct 30 21:20:08.227448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:08.213357    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"
Oct 30 21:20:10.274523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:10.274486    4918 generic.go:356] "Generic (PLEG): container finished" podID="97a16952-efea-4b33-a112-c4b533c353c9" containerID="a51a86fc68a8f6d509a9a9aea97fb6a579cb2f0c2b998464340ccd3461089916" exitCode=0
Oct 30 21:20:10.274979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:10.274556    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" event={"ID":"97a16952-efea-4b33-a112-c4b533c353c9","Type":"ContainerDied","Data":"a51a86fc68a8f6d509a9a9aea97fb6a579cb2f0c2b998464340ccd3461089916"}
Oct 30 21:20:10.275028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:10.274993    4918 scope.go:122] "RemoveContainer" containerID="a51a86fc68a8f6d509a9a9aea97fb6a579cb2f0c2b998464340ccd3461089916"
Oct 30 21:20:10.276723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:10.276684    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:10.276858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:10.276764    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:10.276858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:10.276801    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:11.260151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.260112    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"]
Oct 30 21:20:11.261080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.261055    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1f2c2d6b-cd9f-4c54-86d2-d897d3056800" containerName="payload-processing"
Oct 30 21:20:11.261080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.261081    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2c2d6b-cd9f-4c54-86d2-d897d3056800" containerName="payload-processing"
Oct 30 21:20:11.261239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.261104    4918 cpu_manager.go:414] "containerMap: removing container" podUID="af040445-e395-46fd-9e3d-a617d0d6ad94" containerName="payload-pre-processing"
Oct 30 21:20:11.261239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.261111    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af040445-e395-46fd-9e3d-a617d0d6ad94" containerName="payload-pre-processing"
Oct 30 21:20:11.261239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.261170    4918 cpu_manager.go:414] "containerMap: removing container" podUID="80944c45-978b-4926-b5ca-24eada1bc4e9" containerName="revoke-keys"
Oct 30 21:20:11.261239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.261178    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80944c45-978b-4926-b5ca-24eada1bc4e9" containerName="revoke-keys"
Oct 30 21:20:11.261366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.261330    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2c2d6b-cd9f-4c54-86d2-d897d3056800" containerName="payload-processing"
Oct 30 21:20:11.261366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.261345    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="af040445-e395-46fd-9e3d-a617d0d6ad94" containerName="payload-pre-processing"
Oct 30 21:20:11.261366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.261360    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="80944c45-978b-4926-b5ca-24eada1bc4e9" containerName="revoke-keys"
Oct 30 21:20:11.266087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.266067    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.272348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.272324    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"]
Oct 30 21:20:11.274302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.274268    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-60c26c-openshift-default-dockercfg-6jllx\""
Oct 30 21:20:11.320202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.320165    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" event={"ID":"97a16952-efea-4b33-a112-c4b533c353c9","Type":"ContainerStarted","Data":"7ec4297f203589c7d2e3ce1009bc4fd9133c61e1cb7dc159151996a76ed55f55"}
Oct 30 21:20:11.387417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.386553    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-envoy\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.387417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.386648    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-podinfo\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.387417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.386773    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-certs\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.387417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.386836    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj4nf\" (UniqueName: \"kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-kube-api-access-zj4nf\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.387417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.386919    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-socket\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.387417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.386960    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-token\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.387417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.387024    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-credential-socket\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.387417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.387133    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-data\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.387417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.387167    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istiod-ca-cert\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.487415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.485755    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:20:11.491416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.489554    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 21:20:11.491416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.489622    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 21:20:11.491416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.490955    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj4nf\" (UniqueName: \"kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-kube-api-access-zj4nf\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.491416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.491049    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-socket\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.491416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.491096    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-token\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.491416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.491162    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-credential-socket\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.491416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.491282    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-data\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.491416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.491313    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istiod-ca-cert\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.491935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.491472    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-envoy\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.491935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.491504    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-podinfo\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.491935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.491618    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-certs\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.495416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.492095    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-certs\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.495416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.492644    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-socket\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.495416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.493094    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-credential-socket\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.510557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.509500    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-data\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.518964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.518309    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istiod-ca-cert\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.541167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.541088    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj4nf\" (UniqueName: \"kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-kube-api-access-zj4nf\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.541743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.541689    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-envoy\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.544250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.544206    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-token\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.550497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.545234    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-podinfo\") pod \"e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") " pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.633255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.625718    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:11.736460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.709036    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"]
Oct 30 21:20:11.736460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.723455    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.756468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.746946    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"]
Oct 30 21:20:11.810424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.792679    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5bc8c60a-openshift-default-dockercfg-nnv8f\""
Oct 30 21:20:11.821433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.818158    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-envoy\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.821433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.818218    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-data\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.821433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.818262    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-token\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.821433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.818351    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istiod-ca-cert\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.821433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.818425    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-podinfo\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.821433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.818465    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-socket\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.821433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.818522    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-credential-socket\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.821433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.818574    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62ld\" (UniqueName: \"kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-kube-api-access-k62ld\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.821433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.818625    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-certs\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.935873    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-data\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.935930    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-token\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.936024    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istiod-ca-cert\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.936078    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-podinfo\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.936116    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-socket\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.936172    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-credential-socket\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.936221    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62ld\" (UniqueName: \"kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-kube-api-access-k62ld\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.936275    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-certs\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.936488    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-envoy\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.940070    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-data\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.940340    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-socket\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.940962    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istiod-ca-cert\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.942004    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-credential-socket\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.944411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.942247    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-certs\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.958679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.957672    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-envoy\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.958679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.958120    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-podinfo\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.962761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.962731    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62ld\" (UniqueName: \"kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-kube-api-access-k62ld\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:11.962923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:11.962818    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-token\") pod \"e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") " pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:12.070657 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:20:12.070630    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0b52b8_a075_4ec6_8ee4_79fec413717d.slice/crio-b09741929f2287fe0a5f33610680cdf917e7672ec1eb9246a941ec5d2b9d77fb WatchSource:0}: Error finding container b09741929f2287fe0a5f33610680cdf917e7672ec1eb9246a941ec5d2b9d77fb: Status 404 returned error can't find the container with id b09741929f2287fe0a5f33610680cdf917e7672ec1eb9246a941ec5d2b9d77fb
Oct 30 21:20:12.072274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.072246    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"]
Oct 30 21:20:12.073293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.073265    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:12.073342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.073325    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:12.073378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.073352    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:12.107412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.105653    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:12.341336 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.341213    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" event={"ID":"5e0b52b8-a075-4ec6-8ee4-79fec413717d","Type":"ContainerStarted","Data":"a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf"}
Oct 30 21:20:12.341955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.341922    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" event={"ID":"5e0b52b8-a075-4ec6-8ee4-79fec413717d","Type":"ContainerStarted","Data":"b09741929f2287fe0a5f33610680cdf917e7672ec1eb9246a941ec5d2b9d77fb"}
Oct 30 21:20:12.364226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.364166    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" podStartSLOduration=1.36414827 podStartE2EDuration="1.36414827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:12.363069542 +0000 UTC m=+1125.409628714" watchObservedRunningTime="2026-10-30 21:20:12.36414827 +0000 UTC m=+1125.410707416"
Oct 30 21:20:12.418706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.418676    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"]
Oct 30 21:20:12.420224 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:20:12.420136    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523e7ae5_fc62_41e4_b4b2_016b9195ec92.slice/crio-7f3f08604ebab67ddb3e90d175a3aa9fe1856257585cf072726b51f652ba081a WatchSource:0}: Error finding container 7f3f08604ebab67ddb3e90d175a3aa9fe1856257585cf072726b51f652ba081a: Status 404 returned error can't find the container with id 7f3f08604ebab67ddb3e90d175a3aa9fe1856257585cf072726b51f652ba081a
Oct 30 21:20:12.422964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.422936    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:12.423024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.422996    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:12.423065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.423026    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:12.486264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.485819    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 21:20:12.486264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.485890    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 21:20:12.637464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:12.627504    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:13.351578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:13.351539    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" event={"ID":"523e7ae5-fc62-41e4-b4b2-016b9195ec92","Type":"ContainerStarted","Data":"b439d8716461ef5f44e3322146c8b5e07a89e5f73f34a98c879a54fac6ef7c01"}
Oct 30 21:20:13.351980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:13.351604    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" event={"ID":"523e7ae5-fc62-41e4-b4b2-016b9195ec92","Type":"ContainerStarted","Data":"7f3f08604ebab67ddb3e90d175a3aa9fe1856257585cf072726b51f652ba081a"}
Oct 30 21:20:13.410895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:13.410832    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" podStartSLOduration=2.410812839 podStartE2EDuration="2.410812839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:13.404907226 +0000 UTC m=+1126.451466371" watchObservedRunningTime="2026-10-30 21:20:13.410812839 +0000 UTC m=+1126.457371984"
Oct 30 21:20:13.476985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:13.476794    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:20:13.484129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:13.484099    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 21:20:13.484434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:13.484405    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 21:20:13.643787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:13.643702    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:14.127417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:14.122876    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:14.127417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:14.126962    4918 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 21:20:14.127417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:14.127016    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 21:20:14.459478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:14.449160    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:14.473414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:14.468651    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:14.483822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:14.483796    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 21:20:14.483962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:14.483843    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 21:20:15.126418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:15.118808    4918 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 21:20:15.126418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:15.118867    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 21:20:15.492417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:15.488184    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 21:20:15.492417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:15.488242    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 21:20:16.108242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:16.107799    4918 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 21:20:16.108242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:16.107863    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 21:20:16.488533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:16.488501    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 21:20:16.556416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:16.502498    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 21:20:17.284417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:17.278149    4918 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 21:20:17.284417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:17.278229    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 21:20:17.484410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:17.484352    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 21:20:17.484621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:17.484429    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 21:20:18.110790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:18.110752    4918 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 21:20:18.111382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:18.111353    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 21:20:18.555916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:18.555866    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:20:18.576925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:18.576892    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:20:19.044687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:19.044663    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"]
Oct 30 21:20:19.050281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:19.050260    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27d0b599-v7c59"]
Oct 30 21:20:19.077955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:19.077928    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff" path="/var/lib/kubelet/pods/09d9481e-d22b-4e57-b2d3-d8edfdb3a1ff/volumes"
Oct 30 21:20:19.115414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:19.114851    4918 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 21:20:19.115414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:19.114921    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 21:20:20.159427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:20.153972    4918 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 30 21:20:20.159427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:20.154042    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 30 21:20:21.604846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:21.604804    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:21.606975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:21.606956    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:21.615479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:21.615454    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:24.534415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:24.530510    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"]
Oct 30 21:20:24.539076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:24.537753    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e494a9-c2bc7"]
Oct 30 21:20:25.078948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:25.078909    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80944c45-978b-4926-b5ca-24eada1bc4e9" path="/var/lib/kubelet/pods/80944c45-978b-4926-b5ca-24eada1bc4e9/volumes"
Oct 30 21:20:27.589194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:27.589160    4918 generic.go:356] "Generic (PLEG): container finished" podID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerID="7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19" exitCode=0
Oct 30 21:20:27.589677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:27.589207    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h" event={"ID":"19c28dba-cda4-44cd-8f7e-ac1774dfe309","Type":"ContainerDied","Data":"7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19"}
Oct 30 21:20:27.589722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:27.589678    4918 scope.go:122] "RemoveContainer" containerID="7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19"
Oct 30 21:20:27.591270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:27.591241    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:27.591433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:27.591326    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:27.591433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:27.591372    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:28.625695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:28.625659    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h" event={"ID":"19c28dba-cda4-44cd-8f7e-ac1774dfe309","Type":"ContainerStarted","Data":"b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b"}
Oct 30 21:20:28.815288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:28.815253    4918 scope.go:122] "RemoveContainer" containerID="ed8b563d1ee2e41e3656606905c7e1a8721855600f1f868f3b745ce1a1407c87"
Oct 30 21:20:28.827228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:28.827207    4918 scope.go:122] "RemoveContainer" containerID="f8762aae30d1afcdea023df1329b23808f670be1ebe60e76f4b37e919ec4cafe"
Oct 30 21:20:28.837625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:28.837602    4918 scope.go:122] "RemoveContainer" containerID="d602fbd04912bf4cef54dfdfd2e19ba89d46a674c13632b836edc3017477c6f5"
Oct 30 21:20:29.034981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:29.034955    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"]
Oct 30 21:20:29.040814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:29.040795    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8334ee8-bprth"]
Oct 30 21:20:29.079077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:29.079047    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0249779b-f7da-435d-ac3d-5c77766db493" path="/var/lib/kubelet/pods/0249779b-f7da-435d-ac3d-5c77766db493/volumes"
Oct 30 21:20:29.162139 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:20:29.162093    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:20:29.419097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:29.419065    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:20:29.419380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:29.419355    4918 patch_prober.go:29] interesting pod/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:20:29.419504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:29.419447    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h" podUID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 21:20:30.424416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:30.424192    4918 patch_prober.go:29] interesting pod/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:20:30.424416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:30.424278    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h" podUID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 21:20:31.114567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:31.114528    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"]
Oct 30 21:20:31.118759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:31.118733    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b43959-fhnn2"]
Oct 30 21:20:31.419209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:31.419126    4918 patch_prober.go:29] interesting pod/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:20:31.419209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:31.419184    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h" podUID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 21:20:32.434421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:32.431518    4918 patch_prober.go:29] interesting pod/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:20:32.434421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:32.431621    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h" podUID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 21:20:32.496904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:32.495887    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"]
Oct 30 21:20:32.496904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:32.496147    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h" podUID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerName="istio-proxy" containerID="cri-o://b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b" gracePeriod=30
Oct 30 21:20:33.114492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:33.113430    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f" path="/var/lib/kubelet/pods/c3eb3725-b31d-4e4c-8c8b-8dc79d4b1b4f/volumes"
Oct 30 21:20:34.033148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:34.033115    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"]
Oct 30 21:20:34.037470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:34.037442    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff66b4f-kp68k"]
Oct 30 21:20:34.565836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:34.565790    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:20:35.135413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:35.130997    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a85fae0-d8ef-4579-806b-54d744619594" path="/var/lib/kubelet/pods/2a85fae0-d8ef-4579-806b-54d744619594/volumes"
Oct 30 21:20:39.022968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.022935    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"]
Oct 30 21:20:39.027149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.027116    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-972c9a4c-67946"]
Oct 30 21:20:39.078405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.078361    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b116033-6890-4cb3-b1b3-ff8e8b25a8c0" path="/var/lib/kubelet/pods/1b116033-6890-4cb3-b1b3-ff8e8b25a8c0/volumes"
Oct 30 21:20:39.805980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.805952    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"]
Oct 30 21:20:39.812127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.812107    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"
Oct 30 21:20:39.816439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.816383    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-43f6ab\""
Oct 30 21:20:39.821118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.821029    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"]
Oct 30 21:20:39.842555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.842530    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"]
Oct 30 21:20:39.847260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.847242    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"
Oct 30 21:20:39.855743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.855718    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-43f6ab-dockercfg-7gc8l\""
Oct 30 21:20:39.866699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.863297    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"]
Oct 30 21:20:39.890868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.890836    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt7tl\" (UniqueName: \"kubernetes.io/projected/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-kube-api-access-lt7tl\") pod \"payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln\" (UID: \"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"
Oct 30 21:20:39.891086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:39.891067    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln\" (UID: \"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"
Oct 30 21:20:40.002447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.002402    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln\" (UID: \"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"
Oct 30 21:20:40.003777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.003753    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt7tl\" (UniqueName: \"kubernetes.io/projected/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-kube-api-access-lt7tl\") pod \"payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln\" (UID: \"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"
Oct 30 21:20:40.004066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.004048    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9\" (UID: \"9a52b55e-ed84-4a7e-9a72-d001e7e116e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"
Oct 30 21:20:40.009479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.004250    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vfch\" (UniqueName: \"kubernetes.io/projected/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-kube-api-access-2vfch\") pod \"payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9\" (UID: \"9a52b55e-ed84-4a7e-9a72-d001e7e116e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"
Oct 30 21:20:40.009850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.003528    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln\" (UID: \"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"
Oct 30 21:20:40.034411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.025716    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt7tl\" (UniqueName: \"kubernetes.io/projected/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-kube-api-access-lt7tl\") pod \"payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln\" (UID: \"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"
Oct 30 21:20:40.114474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.113677    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9\" (UID: \"9a52b55e-ed84-4a7e-9a72-d001e7e116e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"
Oct 30 21:20:40.114474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.113827    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vfch\" (UniqueName: \"kubernetes.io/projected/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-kube-api-access-2vfch\") pod \"payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9\" (UID: \"9a52b55e-ed84-4a7e-9a72-d001e7e116e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"
Oct 30 21:20:40.129411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.115408    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9\" (UID: \"9a52b55e-ed84-4a7e-9a72-d001e7e116e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"
Oct 30 21:20:40.129411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.126373    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vfch\" (UniqueName: \"kubernetes.io/projected/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-kube-api-access-2vfch\") pod \"payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9\" (UID: \"9a52b55e-ed84-4a7e-9a72-d001e7e116e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"
Oct 30 21:20:40.148870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.130665    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"
Oct 30 21:20:40.220062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.220024    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"
Oct 30 21:20:40.630105 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.630081    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:20:40.733286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.733258    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-podinfo\") pod \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") "
Oct 30 21:20:40.733491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.733295    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-certs\") pod \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") "
Oct 30 21:20:40.733491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.733346    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istiod-ca-cert\") pod \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") "
Oct 30 21:20:40.733491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.733483    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-token\" (UniqueName: \"kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-token\") pod \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") "
Oct 30 21:20:40.733682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.733602    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-socket\") pod \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") "
Oct 30 21:20:40.733682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.733666    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-certs" pod "19c28dba-cda4-44cd-8f7e-ac1774dfe309" (UID: "19c28dba-cda4-44cd-8f7e-ac1774dfe309"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:40.733809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.733694    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-kube-api-access-w6qf9\" (UniqueName: \"kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-kube-api-access-w6qf9\") pod \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") "
Oct 30 21:20:40.733809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.733757    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-data\") pod \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") "
Oct 30 21:20:40.733916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.733840    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-envoy\") pod \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") "
Oct 30 21:20:40.733916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.733869    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-socket" pod "19c28dba-cda4-44cd-8f7e-ac1774dfe309" (UID: "19c28dba-cda4-44cd-8f7e-ac1774dfe309"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:40.733916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.733891    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istiod-ca-cert" pod "19c28dba-cda4-44cd-8f7e-ac1774dfe309" (UID: "19c28dba-cda4-44cd-8f7e-ac1774dfe309"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:40.733916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.733903    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-credential-socket\") pod \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\" (UID: \"19c28dba-cda4-44cd-8f7e-ac1774dfe309\") "
Oct 30 21:20:40.734299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.734011    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-data" pod "19c28dba-cda4-44cd-8f7e-ac1774dfe309" (UID: "19c28dba-cda4-44cd-8f7e-ac1774dfe309"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:40.734299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.734247    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-credential-socket" pod "19c28dba-cda4-44cd-8f7e-ac1774dfe309" (UID: "19c28dba-cda4-44cd-8f7e-ac1774dfe309"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:40.734896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.734873    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:40.735016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.734900    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:40.735016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.734914    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:40.735016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.734928    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:40.735016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.734944    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:40.735818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.735788    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-podinfo" pod "19c28dba-cda4-44cd-8f7e-ac1774dfe309" (UID: "19c28dba-cda4-44cd-8f7e-ac1774dfe309"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:20:40.735923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.735869    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-token" pod "19c28dba-cda4-44cd-8f7e-ac1774dfe309" (UID: "19c28dba-cda4-44cd-8f7e-ac1774dfe309"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:40.736022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.736009    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-envoy" pod "19c28dba-cda4-44cd-8f7e-ac1774dfe309" (UID: "19c28dba-cda4-44cd-8f7e-ac1774dfe309"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:40.736140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.736122    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-kube-api-access-w6qf9" pod "19c28dba-cda4-44cd-8f7e-ac1774dfe309" (UID: "19c28dba-cda4-44cd-8f7e-ac1774dfe309"). InnerVolumeSpecName "kube-api-access-w6qf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:40.746322 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:20:40.746296    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da9d34e_0952_4f38_8fe3_ea196e8dc3b9.slice/crio-2d8c1bd517206412f8cb37b4fe574daf9185e711a0f3820443557c1dc2dde475 WatchSource:0}: Error finding container 2d8c1bd517206412f8cb37b4fe574daf9185e711a0f3820443557c1dc2dde475: Status 404 returned error can't find the container with id 2d8c1bd517206412f8cb37b4fe574daf9185e711a0f3820443557c1dc2dde475
Oct 30 21:20:40.747510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.747489    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"]
Oct 30 21:20:40.748300 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:20:40.748059    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a52b55e_ed84_4a7e_9a72_d001e7e116e7.slice/crio-b35c04e7a34af9924ca921388d3fda6d835c8ade914c961b8c906e5ba8c6f2d8 WatchSource:0}: Error finding container b35c04e7a34af9924ca921388d3fda6d835c8ade914c961b8c906e5ba8c6f2d8: Status 404 returned error can't find the container with id b35c04e7a34af9924ca921388d3fda6d835c8ade914c961b8c906e5ba8c6f2d8
Oct 30 21:20:40.752701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.752681    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"]
Oct 30 21:20:40.836791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.836747    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:40.836791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.836783    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6qf9\" (UniqueName: \"kubernetes.io/projected/19c28dba-cda4-44cd-8f7e-ac1774dfe309-kube-api-access-w6qf9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:40.837032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.836799    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:40.837032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.836812    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c28dba-cda4-44cd-8f7e-ac1774dfe309-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:40.925979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.925939    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9" event={"ID":"9a52b55e-ed84-4a7e-9a72-d001e7e116e7","Type":"ContainerStarted","Data":"70fd3096355550ff7a22d4f32921af418b8e891a3005bbf163dc78e1ebc27ba6"}
Oct 30 21:20:40.926176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.926038    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9" event={"ID":"9a52b55e-ed84-4a7e-9a72-d001e7e116e7","Type":"ContainerStarted","Data":"b35c04e7a34af9924ca921388d3fda6d835c8ade914c961b8c906e5ba8c6f2d8"}
Oct 30 21:20:40.926445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.926426    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"
Oct 30 21:20:40.928893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.928864    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln" event={"ID":"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9","Type":"ContainerStarted","Data":"7fe6a72555e9965cbf941884b8aea945f3580fcf6fbb701042233e6365ce1113"}
Oct 30 21:20:40.928999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.928909    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln" event={"ID":"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9","Type":"ContainerStarted","Data":"2d8c1bd517206412f8cb37b4fe574daf9185e711a0f3820443557c1dc2dde475"}
Oct 30 21:20:40.928999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.928962    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"
Oct 30 21:20:40.931545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.931520    4918 generic.go:356] "Generic (PLEG): container finished" podID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerID="b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b" exitCode=0
Oct 30 21:20:40.931671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.931570    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"
Oct 30 21:20:40.931671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.931576    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h" event={"ID":"19c28dba-cda4-44cd-8f7e-ac1774dfe309","Type":"ContainerDied","Data":"b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b"}
Oct 30 21:20:40.931671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.931618    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h" event={"ID":"19c28dba-cda4-44cd-8f7e-ac1774dfe309","Type":"ContainerDied","Data":"a8a97a346ea9c8e198ed90be3163c4481d66fbc3c6bdbdd19ea119a2250c80f8"}
Oct 30 21:20:40.931671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.931647    4918 scope.go:122] "RemoveContainer" containerID="b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b"
Oct 30 21:20:40.944735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.944720    4918 scope.go:122] "RemoveContainer" containerID="7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19"
Oct 30 21:20:40.949282 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.949242    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9" podStartSLOduration=1.9492212599999998 podStartE2EDuration="1.94922126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:40.945171105 +0000 UTC m=+1153.991730260" watchObservedRunningTime="2026-10-30 21:20:40.94922126 +0000 UTC m=+1153.995780404"
Oct 30 21:20:40.958331 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.958314    4918 scope.go:122] "RemoveContainer" containerID="b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b"
Oct 30 21:20:40.958665 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:20:40.958636    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b\": container with ID starting with b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b not found: ID does not exist" containerID="b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b"
Oct 30 21:20:40.958738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.958677    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b"} err="failed to get container status \"b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b\": rpc error: code = NotFound desc = could not find container \"b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b\": container with ID starting with b92d81d795f6b0ad334badff5965408d5ba55f507708d4430ecaec2ece69818b not found: ID does not exist"
Oct 30 21:20:40.958738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.958694    4918 scope.go:122] "RemoveContainer" containerID="7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19"
Oct 30 21:20:40.958930 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:20:40.958911    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19\": container with ID starting with 7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19 not found: ID does not exist" containerID="7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19"
Oct 30 21:20:40.958988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.958940    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19"} err="failed to get container status \"7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19\": rpc error: code = NotFound desc = could not find container \"7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19\": container with ID starting with 7bb2b464e1c63c01c5479faabb355e4703e8e551b295a50a227a8378f8b58c19 not found: ID does not exist"
Oct 30 21:20:40.972037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.971990    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln" podStartSLOduration=1.9719749690000001 podStartE2EDuration="1.971974969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:40.966546471 +0000 UTC m=+1154.013105628" watchObservedRunningTime="2026-10-30 21:20:40.971974969 +0000 UTC m=+1154.018534119"
Oct 30 21:20:40.987006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.986983    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"]
Oct 30 21:20:40.991045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:40.991024    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e494a9-openshift-default-5646cb97f5-4p42h"]
Oct 30 21:20:41.080816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:41.080741    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" path="/var/lib/kubelet/pods/19c28dba-cda4-44cd-8f7e-ac1774dfe309/volumes"
Oct 30 21:20:42.669765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:42.669737    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"]
Oct 30 21:20:42.670623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:42.670601    4918 cpu_manager.go:414] "containerMap: removing container" podUID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerName="istio-proxy"
Oct 30 21:20:42.670623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:42.670623    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerName="istio-proxy"
Oct 30 21:20:42.670847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:42.670833    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerName="istio-proxy"
Oct 30 21:20:42.670907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:42.670854    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerName="istio-proxy"
Oct 30 21:20:42.675927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:42.675906    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"
Oct 30 21:20:42.683759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:42.683730    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-60c26c-dockercfg-rt2qf\""
Oct 30 21:20:42.683922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:42.683882    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-60c26c\""
Oct 30 21:20:42.684817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:42.684796    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"]
Oct 30 21:20:42.864505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:42.864460    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc77c6a-4698-4ab3-bb08-d44b211720c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v\" (UID: \"9dc77c6a-4698-4ab3-bb08-d44b211720c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"
Oct 30 21:20:42.892795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:42.892753    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bffqt\" (UniqueName: \"kubernetes.io/projected/9dc77c6a-4698-4ab3-bb08-d44b211720c8-kube-api-access-bffqt\") pod \"payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v\" (UID: \"9dc77c6a-4698-4ab3-bb08-d44b211720c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"
Oct 30 21:20:43.032646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:43.032479    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc77c6a-4698-4ab3-bb08-d44b211720c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v\" (UID: \"9dc77c6a-4698-4ab3-bb08-d44b211720c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"
Oct 30 21:20:43.032859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:43.032704    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffqt\" (UniqueName: \"kubernetes.io/projected/9dc77c6a-4698-4ab3-bb08-d44b211720c8-kube-api-access-bffqt\") pod \"payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v\" (UID: \"9dc77c6a-4698-4ab3-bb08-d44b211720c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"
Oct 30 21:20:43.088468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:43.082808    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc77c6a-4698-4ab3-bb08-d44b211720c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v\" (UID: \"9dc77c6a-4698-4ab3-bb08-d44b211720c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"
Oct 30 21:20:43.105485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:43.101926    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bffqt\" (UniqueName: \"kubernetes.io/projected/9dc77c6a-4698-4ab3-bb08-d44b211720c8-kube-api-access-bffqt\") pod \"payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v\" (UID: \"9dc77c6a-4698-4ab3-bb08-d44b211720c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"
Oct 30 21:20:43.308494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:43.305673    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"
Oct 30 21:20:43.868633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:43.868160    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"]
Oct 30 21:20:44.207518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:44.205281    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v" event={"ID":"9dc77c6a-4698-4ab3-bb08-d44b211720c8","Type":"ContainerStarted","Data":"4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91"}
Oct 30 21:20:44.207518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:44.205409    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v" event={"ID":"9dc77c6a-4698-4ab3-bb08-d44b211720c8","Type":"ContainerStarted","Data":"431a83738869a7469345ea6402c3fef088bf75a0392bd3d2c56d2fcf5f5e24e1"}
Oct 30 21:20:44.212527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:44.212497    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"
Oct 30 21:20:44.256537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:44.256478    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v" podStartSLOduration=2.256459921 podStartE2EDuration="2.256459921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:20:44.25190259 +0000 UTC m=+1157.298461737" watchObservedRunningTime="2026-10-30 21:20:44.256459921 +0000 UTC m=+1157.303019068"
Oct 30 21:20:50.927752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:50.927716    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"]
Oct 30 21:20:50.928181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:50.927985    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerName="istio-proxy" containerID="cri-o://b439d8716461ef5f44e3322146c8b5e07a89e5f73f34a98c879a54fac6ef7c01" gracePeriod=30
Oct 30 21:20:51.609144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:51.609103    4918 patch_prober.go:29] interesting pod/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:20:51.609342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:51.609182    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:20:51.945860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:51.945784    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"
Oct 30 21:20:51.946248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:51.946179    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"
Oct 30 21:20:54.058457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:54.056902    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"]
Oct 30 21:20:54.124852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:54.124733    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53582c8-7b6gl"]
Oct 30 21:20:54.444839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:54.444757    4918 generic.go:356] "Generic (PLEG): container finished" podID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerID="a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf" exitCode=0
Oct 30 21:20:54.444992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:54.444832    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" event={"ID":"5e0b52b8-a075-4ec6-8ee4-79fec413717d","Type":"ContainerDied","Data":"a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf"}
Oct 30 21:20:54.445281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:54.445266    4918 scope.go:122] "RemoveContainer" containerID="a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf"
Oct 30 21:20:54.446854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:54.446827    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:54.446944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:54.446895    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:54.446944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:54.446928    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:20:55.079456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:55.079430    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18fcc7dd-87e0-4995-82bd-a6b55ed5dd95" path="/var/lib/kubelet/pods/18fcc7dd-87e0-4995-82bd-a6b55ed5dd95/volumes"
Oct 30 21:20:55.231366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:55.231338    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"
Oct 30 21:20:55.457348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:55.457278    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" event={"ID":"5e0b52b8-a075-4ec6-8ee4-79fec413717d","Type":"ContainerStarted","Data":"9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0"}
Oct 30 21:20:55.626537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:55.626510    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:55.627602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:55.627574    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:20:55.627738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:55.627623    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 21:20:56.049416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:56.048440    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"]
Oct 30 21:20:56.064967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:56.064934    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8402af68-2zbcn"]
Oct 30 21:20:56.754470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:56.753545    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:20:56.754470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:56.753616    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 21:20:56.888416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:56.880773    4918 generic.go:356] "Generic (PLEG): container finished" podID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerID="b439d8716461ef5f44e3322146c8b5e07a89e5f73f34a98c879a54fac6ef7c01" exitCode=0
Oct 30 21:20:56.888416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:56.882119    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" event={"ID":"523e7ae5-fc62-41e4-b4b2-016b9195ec92","Type":"ContainerDied","Data":"b439d8716461ef5f44e3322146c8b5e07a89e5f73f34a98c879a54fac6ef7c01"}
Oct 30 21:20:56.977212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:56.964039    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:57.084514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.065339    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-token\" (UniqueName: \"kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-token\") pod \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") "
Oct 30 21:20:57.084869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.084839    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-data\") pod \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") "
Oct 30 21:20:57.220424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.092871    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-token" pod "523e7ae5-fc62-41e4-b4b2-016b9195ec92" (UID: "523e7ae5-fc62-41e4-b4b2-016b9195ec92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:57.220675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.131507    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-data" pod "523e7ae5-fc62-41e4-b4b2-016b9195ec92" (UID: "523e7ae5-fc62-41e4-b4b2-016b9195ec92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.220769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.199705    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-envoy\") pod \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") "
Oct 30 21:20:57.220974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.220956    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-socket\") pod \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") "
Oct 30 21:20:57.246416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.221148    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-certs\") pod \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") "
Oct 30 21:20:57.246724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.246700    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-credential-socket\") pod \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") "
Oct 30 21:20:57.246863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.246851    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istiod-ca-cert\") pod \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") "
Oct 30 21:20:57.247001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.246989    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-podinfo\") pod \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") "
Oct 30 21:20:57.264860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.260523    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-kube-api-access-k62ld\" (UniqueName: \"kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-kube-api-access-k62ld\") pod \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\" (UID: \"523e7ae5-fc62-41e4-b4b2-016b9195ec92\") "
Oct 30 21:20:57.264860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.229188    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-certs" pod "523e7ae5-fc62-41e4-b4b2-016b9195ec92" (UID: "523e7ae5-fc62-41e4-b4b2-016b9195ec92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.264860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.238539    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-socket" pod "523e7ae5-fc62-41e4-b4b2-016b9195ec92" (UID: "523e7ae5-fc62-41e4-b4b2-016b9195ec92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.264860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.259685    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-credential-socket" pod "523e7ae5-fc62-41e4-b4b2-016b9195ec92" (UID: "523e7ae5-fc62-41e4-b4b2-016b9195ec92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.264860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.259968    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istiod-ca-cert" pod "523e7ae5-fc62-41e4-b4b2-016b9195ec92" (UID: "523e7ae5-fc62-41e4-b4b2-016b9195ec92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:20:57.264860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.245036    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a56a6e36-5652-409e-b2f0-e912f657c8c6" path="/var/lib/kubelet/pods/a56a6e36-5652-409e-b2f0-e912f657c8c6/volumes"
Oct 30 21:20:57.264860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.261557    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.264860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.261580    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.264860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.261596    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.264860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.261610    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.264860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.261623    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.264860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.261640    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.286822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.276032    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-envoy" pod "523e7ae5-fc62-41e4-b4b2-016b9195ec92" (UID: "523e7ae5-fc62-41e4-b4b2-016b9195ec92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:20:57.286822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.286077    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-kube-api-access-k62ld" pod "523e7ae5-fc62-41e4-b4b2-016b9195ec92" (UID: "523e7ae5-fc62-41e4-b4b2-016b9195ec92"). InnerVolumeSpecName "kube-api-access-k62ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:20:57.286822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.286240    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-podinfo" pod "523e7ae5-fc62-41e4-b4b2-016b9195ec92" (UID: "523e7ae5-fc62-41e4-b4b2-016b9195ec92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:20:57.363060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.362986    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.363060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.363012    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/523e7ae5-fc62-41e4-b4b2-016b9195ec92-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.363060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.363023    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62ld\" (UniqueName: \"kubernetes.io/projected/523e7ae5-fc62-41e4-b4b2-016b9195ec92-kube-api-access-k62ld\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:20:57.627676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.627591    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:20:57.627819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.627667    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 21:20:57.916330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.916249    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"
Oct 30 21:20:57.916330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.916284    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp" event={"ID":"523e7ae5-fc62-41e4-b4b2-016b9195ec92","Type":"ContainerDied","Data":"7f3f08604ebab67ddb3e90d175a3aa9fe1856257585cf072726b51f652ba081a"}
Oct 30 21:20:57.916900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.916354    4918 scope.go:122] "RemoveContainer" containerID="b439d8716461ef5f44e3322146c8b5e07a89e5f73f34a98c879a54fac6ef7c01"
Oct 30 21:20:57.917230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.917205    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"]
Oct 30 21:20:57.917992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.917975    4918 cpu_manager.go:414] "containerMap: removing container" podUID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerName="istio-proxy"
Oct 30 21:20:57.918064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.917995    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c28dba-cda4-44cd-8f7e-ac1774dfe309" containerName="istio-proxy"
Oct 30 21:20:57.918064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.918046    4918 cpu_manager.go:414] "containerMap: removing container" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerName="istio-proxy"
Oct 30 21:20:57.918064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.918051    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerName="istio-proxy"
Oct 30 21:20:57.918495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.918160    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" containerName="istio-proxy"
Oct 30 21:20:57.923217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.923201    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"
Oct 30 21:20:57.929937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.929914    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"]
Oct 30 21:20:57.930909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.930886    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dbc08f-dockercfg-5l92n\""
Oct 30 21:20:57.931014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.930916    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dbc08f\""
Oct 30 21:20:57.947025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.947001    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"]
Oct 30 21:20:57.951030 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.951003    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5bc8c60a-openshift-default-67988b845-cv9dp"]
Oct 30 21:20:57.962161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.962138    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"]
Oct 30 21:20:57.966513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.966492    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"
Oct 30 21:20:57.970711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.970691    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1829fc-690b-4568-9a52-3e0b5344cd26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv\" (UID: \"fb1829fc-690b-4568-9a52-3e0b5344cd26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"
Oct 30 21:20:57.971013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.970992    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqb9g\" (UniqueName: \"kubernetes.io/projected/fb1829fc-690b-4568-9a52-3e0b5344cd26-kube-api-access-jqb9g\") pod \"payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv\" (UID: \"fb1829fc-690b-4568-9a52-3e0b5344cd26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"
Oct 30 21:20:57.984095 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:57.984070    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"]
Oct 30 21:20:58.083442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.074294    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7\" (UID: \"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af\") " pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"
Oct 30 21:20:58.083442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.074372    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqb9g\" (UniqueName: \"kubernetes.io/projected/fb1829fc-690b-4568-9a52-3e0b5344cd26-kube-api-access-jqb9g\") pod \"payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv\" (UID: \"fb1829fc-690b-4568-9a52-3e0b5344cd26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"
Oct 30 21:20:58.083442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.074498    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1829fc-690b-4568-9a52-3e0b5344cd26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv\" (UID: \"fb1829fc-690b-4568-9a52-3e0b5344cd26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"
Oct 30 21:20:58.083442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.074541    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjhj5\" (UniqueName: \"kubernetes.io/projected/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-kube-api-access-fjhj5\") pod \"payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7\" (UID: \"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af\") " pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"
Oct 30 21:20:58.083442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.075897    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1829fc-690b-4568-9a52-3e0b5344cd26-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv\" (UID: \"fb1829fc-690b-4568-9a52-3e0b5344cd26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"
Oct 30 21:20:58.160943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.160903    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqb9g\" (UniqueName: \"kubernetes.io/projected/fb1829fc-690b-4568-9a52-3e0b5344cd26-kube-api-access-jqb9g\") pod \"payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv\" (UID: \"fb1829fc-690b-4568-9a52-3e0b5344cd26\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"
Oct 30 21:20:58.194444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.182513    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjhj5\" (UniqueName: \"kubernetes.io/projected/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-kube-api-access-fjhj5\") pod \"payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7\" (UID: \"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af\") " pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"
Oct 30 21:20:58.194444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.182822    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7\" (UID: \"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af\") " pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"
Oct 30 21:20:58.194444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.185147    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7\" (UID: \"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af\") " pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"
Oct 30 21:20:58.220417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.217908    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjhj5\" (UniqueName: \"kubernetes.io/projected/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-kube-api-access-fjhj5\") pod \"payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7\" (UID: \"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af\") " pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"
Oct 30 21:20:58.256677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.256348    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"
Oct 30 21:20:58.317196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.303156    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"
Oct 30 21:20:58.629131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.629075    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:20:58.683299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:58.629469    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 21:20:59.167138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:59.165008    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523e7ae5-fc62-41e4-b4b2-016b9195ec92" path="/var/lib/kubelet/pods/523e7ae5-fc62-41e4-b4b2-016b9195ec92/volumes"
Oct 30 21:20:59.167138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:59.165625    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"]
Oct 30 21:20:59.167138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:59.165773    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3caf3a-s6zmn"]
Oct 30 21:20:59.308809 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:20:59.308611    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:20:59.417509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:59.409708    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"]
Oct 30 21:20:59.417509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:59.411520    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"]
Oct 30 21:20:59.449962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:59.449902    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:20:59.626523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:59.626490    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:20:59.626703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:20:59.626541    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 21:21:00.165443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:00.165411    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7" event={"ID":"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af","Type":"ContainerStarted","Data":"3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe"}
Oct 30 21:21:00.165642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:00.165464    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7" event={"ID":"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af","Type":"ContainerStarted","Data":"aa8bbc5567ecce293182db122332fd1f659c341363ec752e653ca80a217a63b3"}
Oct 30 21:21:00.165642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:00.165516    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"
Oct 30 21:21:00.167270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:00.167244    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv" event={"ID":"fb1829fc-690b-4568-9a52-3e0b5344cd26","Type":"ContainerStarted","Data":"8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343"}
Oct 30 21:21:00.167696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:00.167282    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv" event={"ID":"fb1829fc-690b-4568-9a52-3e0b5344cd26","Type":"ContainerStarted","Data":"0b4b4423f9acec4ca52510804aac554ccf79b7c73df3dc870d9ee421bb32005e"}
Oct 30 21:21:00.167696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:00.167308    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"
Oct 30 21:21:00.183644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:00.183592    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7" podStartSLOduration=3.183580801 podStartE2EDuration="3.183580801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:00.18278064 +0000 UTC m=+1173.229339785" watchObservedRunningTime="2026-10-30 21:21:00.183580801 +0000 UTC m=+1173.230139945"
Oct 30 21:21:00.227140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:00.227084    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv" podStartSLOduration=3.227066458 podStartE2EDuration="3.227066458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:00.220851622 +0000 UTC m=+1173.267410768" watchObservedRunningTime="2026-10-30 21:21:00.227066458 +0000 UTC m=+1173.273625604"
Oct 30 21:21:00.627150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:00.627116    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:21:00.627310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:00.627163    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 21:21:01.079168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:01.079137    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5212a51-1dab-44ae-9c46-2ca807ab7143" path="/var/lib/kubelet/pods/f5212a51-1dab-44ae-9c46-2ca807ab7143/volumes"
Oct 30 21:21:01.626770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:01.626725    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Oct 30 21:21:01.627275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:01.626802    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Oct 30 21:21:02.429039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.429004    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"]
Oct 30 21:21:02.433767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.433751    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:02.438372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.438345    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:21:02.438644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.438619    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:21:02.439680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.439659    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:21:02.439771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.439724    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:21:02.441600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.441577    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"]
Oct 30 21:21:02.515279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.508419    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb\" (UID: \"f83349e3-b517-45f0-840d-b431157b6f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:02.515279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.508547    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96ktl\" (UniqueName: \"kubernetes.io/projected/f83349e3-b517-45f0-840d-b431157b6f3c-kube-api-access-96ktl\") pod \"maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb\" (UID: \"f83349e3-b517-45f0-840d-b431157b6f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:02.515279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.508666    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb\" (UID: \"f83349e3-b517-45f0-840d-b431157b6f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:02.609945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.609914    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb\" (UID: \"f83349e3-b517-45f0-840d-b431157b6f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:02.610143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.609986    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96ktl\" (UniqueName: \"kubernetes.io/projected/f83349e3-b517-45f0-840d-b431157b6f3c-kube-api-access-96ktl\") pod \"maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb\" (UID: \"f83349e3-b517-45f0-840d-b431157b6f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:02.610143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.610057    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb\" (UID: \"f83349e3-b517-45f0-840d-b431157b6f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:02.610756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.610736    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb\" (UID: \"f83349e3-b517-45f0-840d-b431157b6f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:02.612544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.612519    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb\" (UID: \"f83349e3-b517-45f0-840d-b431157b6f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:02.618718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.618694    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96ktl\" (UniqueName: \"kubernetes.io/projected/f83349e3-b517-45f0-840d-b431157b6f3c-kube-api-access-96ktl\") pod \"maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb\" (UID: \"f83349e3-b517-45f0-840d-b431157b6f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:02.631139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.631120    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:21:02.632270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.632253    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:21:02.845648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:02.845618    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:03.129649 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:21:03.129621    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83349e3_b517_45f0_840d_b431157b6f3c.slice/crio-2b786babbb99b03880f0137f40633081cd7f866c1250690246c6224bb97d4581 WatchSource:0}: Error finding container 2b786babbb99b03880f0137f40633081cd7f866c1250690246c6224bb97d4581: Status 404 returned error can't find the container with id 2b786babbb99b03880f0137f40633081cd7f866c1250690246c6224bb97d4581
Oct 30 21:21:03.129857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:03.129773    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"]
Oct 30 21:21:03.194901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:03.194875    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb" event={"ID":"f83349e3-b517-45f0-840d-b431157b6f3c","Type":"ContainerStarted","Data":"2b786babbb99b03880f0137f40633081cd7f866c1250690246c6224bb97d4581"}
Oct 30 21:21:04.204693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:04.204660    4918 generic.go:356] "Generic (PLEG): container finished" podID="f83349e3-b517-45f0-840d-b431157b6f3c" containerID="cdb8146b1eea4c0576c4a129bcccff8eb82c7f75fb2547a04083348a4f776262" exitCode=0
Oct 30 21:21:04.205131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:04.204702    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb" event={"ID":"f83349e3-b517-45f0-840d-b431157b6f3c","Type":"ContainerDied","Data":"cdb8146b1eea4c0576c4a129bcccff8eb82c7f75fb2547a04083348a4f776262"}
Oct 30 21:21:05.488504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:05.488478    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:05.598413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:05.598353    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-openshift-ca\") pod \"f83349e3-b517-45f0-840d-b431157b6f3c\" (UID: \"f83349e3-b517-45f0-840d-b431157b6f3c\") "
Oct 30 21:21:05.598574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:05.598440    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-xks-ca\") pod \"f83349e3-b517-45f0-840d-b431157b6f3c\" (UID: \"f83349e3-b517-45f0-840d-b431157b6f3c\") "
Oct 30 21:21:05.598617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:05.598580    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83349e3-b517-45f0-840d-b431157b6f3c-kube-api-access-96ktl\" (UniqueName: \"kubernetes.io/projected/f83349e3-b517-45f0-840d-b431157b6f3c-kube-api-access-96ktl\") pod \"f83349e3-b517-45f0-840d-b431157b6f3c\" (UID: \"f83349e3-b517-45f0-840d-b431157b6f3c\") "
Oct 30 21:21:05.598726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:05.598702    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-openshift-ca" pod "f83349e3-b517-45f0-840d-b431157b6f3c" (UID: "f83349e3-b517-45f0-840d-b431157b6f3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:05.599279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:05.599258    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:05.600722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:05.600696    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83349e3-b517-45f0-840d-b431157b6f3c-kube-api-access-96ktl" pod "f83349e3-b517-45f0-840d-b431157b6f3c" (UID: "f83349e3-b517-45f0-840d-b431157b6f3c"). InnerVolumeSpecName "kube-api-access-96ktl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:05.600828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:05.600718    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-xks-ca" pod "f83349e3-b517-45f0-840d-b431157b6f3c" (UID: "f83349e3-b517-45f0-840d-b431157b6f3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:21:05.716464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:05.707859    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96ktl\" (UniqueName: \"kubernetes.io/projected/f83349e3-b517-45f0-840d-b431157b6f3c-kube-api-access-96ktl\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:05.716464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:05.707921    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83349e3-b517-45f0-840d-b431157b6f3c-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:06.235087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:06.235035    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb" event={"ID":"f83349e3-b517-45f0-840d-b431157b6f3c","Type":"ContainerDied","Data":"2b786babbb99b03880f0137f40633081cd7f866c1250690246c6224bb97d4581"}
Oct 30 21:21:06.235352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:06.235333    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b786babbb99b03880f0137f40633081cd7f866c1250690246c6224bb97d4581"
Oct 30 21:21:06.235533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:06.235520    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"
Oct 30 21:21:07.598273 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.598231    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"]
Oct 30 21:21:07.598973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.598959    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f83349e3-b517-45f0-840d-b431157b6f3c" containerName="revoke-keys"
Oct 30 21:21:07.599014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.598976    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83349e3-b517-45f0-840d-b431157b6f3c" containerName="revoke-keys"
Oct 30 21:21:07.599101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.599092    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83349e3-b517-45f0-840d-b431157b6f3c" containerName="revoke-keys"
Oct 30 21:21:07.602756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.602738    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"
Oct 30 21:21:07.611908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.611888    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2a55fb\""
Oct 30 21:21:07.612986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.612970    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2a55fb-dockercfg-rltxx\""
Oct 30 21:21:07.614679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.614658    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"]
Oct 30 21:21:07.640553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.640521    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"]
Oct 30 21:21:07.647121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.647083    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"
Oct 30 21:21:07.651203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.651177    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"]
Oct 30 21:21:07.784422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.769872    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69flz\" (UniqueName: \"kubernetes.io/projected/4094667e-929d-45e3-a0ca-4167b3895ca3-kube-api-access-69flz\") pod \"payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt\" (UID: \"4094667e-929d-45e3-a0ca-4167b3895ca3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"
Oct 30 21:21:07.808414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.799553    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4094667e-929d-45e3-a0ca-4167b3895ca3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt\" (UID: \"4094667e-929d-45e3-a0ca-4167b3895ca3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"
Oct 30 21:21:07.969106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.969011    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0985b700-2e5c-4adc-84f9-3102272a76ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg\" (UID: \"0985b700-2e5c-4adc-84f9-3102272a76ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"
Oct 30 21:21:07.972037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:07.972007    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28d2v\" (UniqueName: \"kubernetes.io/projected/0985b700-2e5c-4adc-84f9-3102272a76ad-kube-api-access-28d2v\") pod \"payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg\" (UID: \"0985b700-2e5c-4adc-84f9-3102272a76ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"
Oct 30 21:21:08.073534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:08.073495    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0985b700-2e5c-4adc-84f9-3102272a76ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg\" (UID: \"0985b700-2e5c-4adc-84f9-3102272a76ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"
Oct 30 21:21:08.073922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:08.073896    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28d2v\" (UniqueName: \"kubernetes.io/projected/0985b700-2e5c-4adc-84f9-3102272a76ad-kube-api-access-28d2v\") pod \"payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg\" (UID: \"0985b700-2e5c-4adc-84f9-3102272a76ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"
Oct 30 21:21:08.074020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:08.073981    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69flz\" (UniqueName: \"kubernetes.io/projected/4094667e-929d-45e3-a0ca-4167b3895ca3-kube-api-access-69flz\") pod \"payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt\" (UID: \"4094667e-929d-45e3-a0ca-4167b3895ca3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"
Oct 30 21:21:08.074072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:08.074025    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4094667e-929d-45e3-a0ca-4167b3895ca3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt\" (UID: \"4094667e-929d-45e3-a0ca-4167b3895ca3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"
Oct 30 21:21:08.074452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:08.074385    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0985b700-2e5c-4adc-84f9-3102272a76ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg\" (UID: \"0985b700-2e5c-4adc-84f9-3102272a76ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"
Oct 30 21:21:08.074822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:08.074800    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4094667e-929d-45e3-a0ca-4167b3895ca3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt\" (UID: \"4094667e-929d-45e3-a0ca-4167b3895ca3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"
Oct 30 21:21:08.082745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:08.082718    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69flz\" (UniqueName: \"kubernetes.io/projected/4094667e-929d-45e3-a0ca-4167b3895ca3-kube-api-access-69flz\") pod \"payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt\" (UID: \"4094667e-929d-45e3-a0ca-4167b3895ca3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"
Oct 30 21:21:08.082916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:08.082895    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28d2v\" (UniqueName: \"kubernetes.io/projected/0985b700-2e5c-4adc-84f9-3102272a76ad-kube-api-access-28d2v\") pod \"payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg\" (UID: \"0985b700-2e5c-4adc-84f9-3102272a76ad\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"
Oct 30 21:21:08.219066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:08.219036    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"
Oct 30 21:21:08.266771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:08.266748    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"
Oct 30 21:21:08.535073 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:21:08.535040    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4094667e_929d_45e3_a0ca_4167b3895ca3.slice/crio-329f33f5a4082a59b8cc492dbf63bbac6d827a5ee761c65c1e498596dbb3fbc1 WatchSource:0}: Error finding container 329f33f5a4082a59b8cc492dbf63bbac6d827a5ee761c65c1e498596dbb3fbc1: Status 404 returned error can't find the container with id 329f33f5a4082a59b8cc492dbf63bbac6d827a5ee761c65c1e498596dbb3fbc1
Oct 30 21:21:08.535522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:08.535504    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"]
Oct 30 21:21:08.567861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:08.567838    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"]
Oct 30 21:21:08.572673 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:21:08.572649    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0985b700_2e5c_4adc_84f9_3102272a76ad.slice/crio-b68e7ed9c3311a268aa7ae9589bd03449f46f6e6b3d8a52da60e327c34e76762 WatchSource:0}: Error finding container b68e7ed9c3311a268aa7ae9589bd03449f46f6e6b3d8a52da60e327c34e76762: Status 404 returned error can't find the container with id b68e7ed9c3311a268aa7ae9589bd03449f46f6e6b3d8a52da60e327c34e76762
Oct 30 21:21:09.024045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:09.024006    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"]
Oct 30 21:21:09.028035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:09.028010    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1626ea67-fl25x"]
Oct 30 21:21:09.078545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:09.078516    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a6aefd-c0c0-4db2-b058-c4560377140f" path="/var/lib/kubelet/pods/74a6aefd-c0c0-4db2-b058-c4560377140f/volumes"
Oct 30 21:21:09.276980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:09.276891    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg" event={"ID":"0985b700-2e5c-4adc-84f9-3102272a76ad","Type":"ContainerStarted","Data":"f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353"}
Oct 30 21:21:09.276980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:09.276958    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg" event={"ID":"0985b700-2e5c-4adc-84f9-3102272a76ad","Type":"ContainerStarted","Data":"b68e7ed9c3311a268aa7ae9589bd03449f46f6e6b3d8a52da60e327c34e76762"}
Oct 30 21:21:09.277183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:09.276987    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"
Oct 30 21:21:09.278920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:09.278897    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt" event={"ID":"4094667e-929d-45e3-a0ca-4167b3895ca3","Type":"ContainerStarted","Data":"3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57"}
Oct 30 21:21:09.279065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:09.278934    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt" event={"ID":"4094667e-929d-45e3-a0ca-4167b3895ca3","Type":"ContainerStarted","Data":"329f33f5a4082a59b8cc492dbf63bbac6d827a5ee761c65c1e498596dbb3fbc1"}
Oct 30 21:21:09.279065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:09.278971    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"
Oct 30 21:21:09.336370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:09.336309    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg" podStartSLOduration=2.336289628 podStartE2EDuration="2.336289628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:09.331214647 +0000 UTC m=+1182.377773821" watchObservedRunningTime="2026-10-30 21:21:09.336289628 +0000 UTC m=+1182.382848777"
Oct 30 21:21:09.353094 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:09.353050    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt" podStartSLOduration=2.35303528 podStartE2EDuration="2.35303528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:09.350914381 +0000 UTC m=+1182.397473545" watchObservedRunningTime="2026-10-30 21:21:09.35303528 +0000 UTC m=+1182.399594424"
Oct 30 21:21:10.068021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:10.062458    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"]
Oct 30 21:21:10.068021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:10.067919    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f386072-kgls5"]
Oct 30 21:21:11.078714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:11.078683    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52459a4-e5af-4ba1-9237-3d4b6ef15540" path="/var/lib/kubelet/pods/d52459a4-e5af-4ba1-9237-3d4b6ef15540/volumes"
Oct 30 21:21:11.179690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:11.179663    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"
Oct 30 21:21:11.179855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:11.179738    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"
Oct 30 21:21:14.028201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:14.028167    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"]
Oct 30 21:21:14.033935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:14.033904    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2981703-q89x9"]
Oct 30 21:21:15.079329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:15.079298    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7501229-9b58-4af7-b99e-ae3d6f3ae599" path="/var/lib/kubelet/pods/f7501229-9b58-4af7-b99e-ae3d6f3ae599/volumes"
Oct 30 21:21:18.959270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:18.959235    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"]
Oct 30 21:21:18.960214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:18.959521    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62" podUID="4405372a-da79-4d92-95a9-9261167ce3dd" containerName="payload-processing" containerID="cri-o://32ce61271d1b41dd8175fdb9a4186c6cbf5cb98d22e28818dec0ae83495fdcbf" gracePeriod=30
Oct 30 21:21:18.970258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:18.970232    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"]
Oct 30 21:21:18.970699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:18.970641    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk" podUID="edebd91a-ce2a-4544-afa3-f1828eb16f8c" containerName="payload-pre-processing" containerID="cri-o://ff779a4b63502dde12933ebd1aabcbf44df5df6c57c9958a2c1bdda3a3ee315b" gracePeriod=30
Oct 30 21:21:19.591567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.591522    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"]
Oct 30 21:21:19.665218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.665179    4918 generic.go:356] "Generic (PLEG): container finished" podID="4405372a-da79-4d92-95a9-9261167ce3dd" containerID="32ce61271d1b41dd8175fdb9a4186c6cbf5cb98d22e28818dec0ae83495fdcbf" exitCode=0
Oct 30 21:21:19.665869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.665834    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62" event={"ID":"4405372a-da79-4d92-95a9-9261167ce3dd","Type":"ContainerDied","Data":"32ce61271d1b41dd8175fdb9a4186c6cbf5cb98d22e28818dec0ae83495fdcbf"}
Oct 30 21:21:19.669085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.669057    4918 generic.go:356] "Generic (PLEG): container finished" podID="edebd91a-ce2a-4544-afa3-f1828eb16f8c" containerID="ff779a4b63502dde12933ebd1aabcbf44df5df6c57c9958a2c1bdda3a3ee315b" exitCode=0
Oct 30 21:21:19.669362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.669343    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk" event={"ID":"edebd91a-ce2a-4544-afa3-f1828eb16f8c","Type":"ContainerDied","Data":"ff779a4b63502dde12933ebd1aabcbf44df5df6c57c9958a2c1bdda3a3ee315b"}
Oct 30 21:21:19.730194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.708908    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3ac6bf-tjbgb"]
Oct 30 21:21:19.790371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.790329    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"
Oct 30 21:21:19.843147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.843073    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"
Oct 30 21:21:19.961384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.961352    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4405372a-da79-4d92-95a9-9261167ce3dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4405372a-da79-4d92-95a9-9261167ce3dd-plugins-config-volume\") pod \"4405372a-da79-4d92-95a9-9261167ce3dd\" (UID: \"4405372a-da79-4d92-95a9-9261167ce3dd\") "
Oct 30 21:21:19.961775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.961540    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4405372a-da79-4d92-95a9-9261167ce3dd-kube-api-access-7hppb\" (UniqueName: \"kubernetes.io/projected/4405372a-da79-4d92-95a9-9261167ce3dd-kube-api-access-7hppb\") pod \"4405372a-da79-4d92-95a9-9261167ce3dd\" (UID: \"4405372a-da79-4d92-95a9-9261167ce3dd\") "
Oct 30 21:21:19.961775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.961638    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edebd91a-ce2a-4544-afa3-f1828eb16f8c-kube-api-access-5zhc7\" (UniqueName: \"kubernetes.io/projected/edebd91a-ce2a-4544-afa3-f1828eb16f8c-kube-api-access-5zhc7\") pod \"edebd91a-ce2a-4544-afa3-f1828eb16f8c\" (UID: \"edebd91a-ce2a-4544-afa3-f1828eb16f8c\") "
Oct 30 21:21:19.961775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.961723    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4405372a-da79-4d92-95a9-9261167ce3dd-plugins-config-volume" pod "4405372a-da79-4d92-95a9-9261167ce3dd" (UID: "4405372a-da79-4d92-95a9-9261167ce3dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:19.961775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.961747    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edebd91a-ce2a-4544-afa3-f1828eb16f8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edebd91a-ce2a-4544-afa3-f1828eb16f8c-plugins-config-volume\") pod \"edebd91a-ce2a-4544-afa3-f1828eb16f8c\" (UID: \"edebd91a-ce2a-4544-afa3-f1828eb16f8c\") "
Oct 30 21:21:19.962166 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.962143    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edebd91a-ce2a-4544-afa3-f1828eb16f8c-plugins-config-volume" pod "edebd91a-ce2a-4544-afa3-f1828eb16f8c" (UID: "edebd91a-ce2a-4544-afa3-f1828eb16f8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:19.962368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.962344    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edebd91a-ce2a-4544-afa3-f1828eb16f8c-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:19.962547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.962373    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4405372a-da79-4d92-95a9-9261167ce3dd-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:19.964650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.964626    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edebd91a-ce2a-4544-afa3-f1828eb16f8c-kube-api-access-5zhc7" pod "edebd91a-ce2a-4544-afa3-f1828eb16f8c" (UID: "edebd91a-ce2a-4544-afa3-f1828eb16f8c"). InnerVolumeSpecName "kube-api-access-5zhc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:19.964744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:19.964649    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4405372a-da79-4d92-95a9-9261167ce3dd-kube-api-access-7hppb" pod "4405372a-da79-4d92-95a9-9261167ce3dd" (UID: "4405372a-da79-4d92-95a9-9261167ce3dd"). InnerVolumeSpecName "kube-api-access-7hppb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:20.072952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.069911    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hppb\" (UniqueName: \"kubernetes.io/projected/4405372a-da79-4d92-95a9-9261167ce3dd-kube-api-access-7hppb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:20.072952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.069951    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zhc7\" (UniqueName: \"kubernetes.io/projected/edebd91a-ce2a-4544-afa3-f1828eb16f8c-kube-api-access-5zhc7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:20.291265 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.291233    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"
Oct 30 21:21:20.291463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.291316    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"
Oct 30 21:21:20.745814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.745736    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"
Oct 30 21:21:20.746228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.746200    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk" event={"ID":"edebd91a-ce2a-4544-afa3-f1828eb16f8c","Type":"ContainerDied","Data":"594bc786f9e1f06a92d8bb17272fe53e36f6f849dd565672116f06249f51fde7"}
Oct 30 21:21:20.757423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.746473    4918 scope.go:122] "RemoveContainer" containerID="ff779a4b63502dde12933ebd1aabcbf44df5df6c57c9958a2c1bdda3a3ee315b"
Oct 30 21:21:20.760029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.759994    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62" event={"ID":"4405372a-da79-4d92-95a9-9261167ce3dd","Type":"ContainerDied","Data":"20d1642b369bcab3a5a4a86ec6985d424384002ab72962071ca90bdaf6fc53b0"}
Oct 30 21:21:20.760284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.760136    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"
Oct 30 21:21:20.789756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.789738    4918 scope.go:122] "RemoveContainer" containerID="32ce61271d1b41dd8175fdb9a4186c6cbf5cb98d22e28818dec0ae83495fdcbf"
Oct 30 21:21:20.804525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.804502    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"]
Oct 30 21:21:20.809339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.809313    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3ac6bf-d96f97944-qnn62"]
Oct 30 21:21:20.819623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.819600    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"]
Oct 30 21:21:20.827753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:20.827727    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3ac6bf-54cb8f8d7f-26rdk"]
Oct 30 21:21:21.127466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:21.125976    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4405372a-da79-4d92-95a9-9261167ce3dd" path="/var/lib/kubelet/pods/4405372a-da79-4d92-95a9-9261167ce3dd/volumes"
Oct 30 21:21:21.127466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:21.126964    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edebd91a-ce2a-4544-afa3-f1828eb16f8c" path="/var/lib/kubelet/pods/edebd91a-ce2a-4544-afa3-f1828eb16f8c/volumes"
Oct 30 21:21:21.127998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:21.127651    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f83349e3-b517-45f0-840d-b431157b6f3c" path="/var/lib/kubelet/pods/f83349e3-b517-45f0-840d-b431157b6f3c/volumes"
Oct 30 21:21:22.030238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:22.030209    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"]
Oct 30 21:21:22.036309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:22.036283    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31570acd-5cf7w"]
Oct 30 21:21:23.078263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:23.078209    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74bd00e5-5f17-467b-952a-0f7ee99258c8" path="/var/lib/kubelet/pods/74bd00e5-5f17-467b-952a-0f7ee99258c8/volumes"
Oct 30 21:21:24.029081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:24.029041    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"]
Oct 30 21:21:24.034402 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:24.034357    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0897c0d-f8jqg"]
Oct 30 21:21:25.202412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:25.193864    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36945330-cd5e-4c05-94e4-5304ac127bb4" path="/var/lib/kubelet/pods/36945330-cd5e-4c05-94e4-5304ac127bb4/volumes"
Oct 30 21:21:26.952765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.952732    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"]
Oct 30 21:21:26.953448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.953427    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4405372a-da79-4d92-95a9-9261167ce3dd" containerName="payload-processing"
Oct 30 21:21:26.953448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.953445    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4405372a-da79-4d92-95a9-9261167ce3dd" containerName="payload-processing"
Oct 30 21:21:26.953632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.953467    4918 cpu_manager.go:414] "containerMap: removing container" podUID="edebd91a-ce2a-4544-afa3-f1828eb16f8c" containerName="payload-pre-processing"
Oct 30 21:21:26.953632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.953472    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edebd91a-ce2a-4544-afa3-f1828eb16f8c" containerName="payload-pre-processing"
Oct 30 21:21:26.953632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.953595    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4405372a-da79-4d92-95a9-9261167ce3dd" containerName="payload-processing"
Oct 30 21:21:26.953632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.953605    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="edebd91a-ce2a-4544-afa3-f1828eb16f8c" containerName="payload-pre-processing"
Oct 30 21:21:26.958558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.958541    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"
Oct 30 21:21:26.966895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.966872    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d87b5f\""
Oct 30 21:21:26.967613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.967563    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d87b5f-dockercfg-8mldc\""
Oct 30 21:21:26.972852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.972826    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"]
Oct 30 21:21:26.982246 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.982222    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"]
Oct 30 21:21:26.987621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.987604    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"
Oct 30 21:21:26.997951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:26.997577    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"]
Oct 30 21:21:27.151436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.140914    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27df2c10-4316-489e-9a7a-19417b131519-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2\" (UID: \"27df2c10-4316-489e-9a7a-19417b131519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"
Oct 30 21:21:27.151436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.140998    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69\" (UID: \"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6\") " pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"
Oct 30 21:21:27.151436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.141103    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqq4t\" (UniqueName: \"kubernetes.io/projected/27df2c10-4316-489e-9a7a-19417b131519-kube-api-access-pqq4t\") pod \"payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2\" (UID: \"27df2c10-4316-489e-9a7a-19417b131519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"
Oct 30 21:21:27.151436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.141223    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkddh\" (UniqueName: \"kubernetes.io/projected/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-kube-api-access-lkddh\") pod \"payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69\" (UID: \"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6\") " pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"
Oct 30 21:21:27.246214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.245148    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27df2c10-4316-489e-9a7a-19417b131519-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2\" (UID: \"27df2c10-4316-489e-9a7a-19417b131519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"
Oct 30 21:21:27.246214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.245212    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69\" (UID: \"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6\") " pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"
Oct 30 21:21:27.246214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.245294    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqq4t\" (UniqueName: \"kubernetes.io/projected/27df2c10-4316-489e-9a7a-19417b131519-kube-api-access-pqq4t\") pod \"payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2\" (UID: \"27df2c10-4316-489e-9a7a-19417b131519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"
Oct 30 21:21:27.246214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.245384    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkddh\" (UniqueName: \"kubernetes.io/projected/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-kube-api-access-lkddh\") pod \"payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69\" (UID: \"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6\") " pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"
Oct 30 21:21:27.265307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.264678    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d87b5f\""
Oct 30 21:21:27.266745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.266719    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27df2c10-4316-489e-9a7a-19417b131519-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2\" (UID: \"27df2c10-4316-489e-9a7a-19417b131519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"
Oct 30 21:21:27.268064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.267972    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69\" (UID: \"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6\") " pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"
Oct 30 21:21:27.349314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.349276    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqq4t\" (UniqueName: \"kubernetes.io/projected/27df2c10-4316-489e-9a7a-19417b131519-kube-api-access-pqq4t\") pod \"payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2\" (UID: \"27df2c10-4316-489e-9a7a-19417b131519\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"
Oct 30 21:21:27.366854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.366825    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkddh\" (UniqueName: \"kubernetes.io/projected/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-kube-api-access-lkddh\") pod \"payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69\" (UID: \"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6\") " pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"
Oct 30 21:21:27.581448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.581354    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d87b5f-dockercfg-8mldc\""
Oct 30 21:21:27.589472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.589447    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"
Oct 30 21:21:27.607111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.607035    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"
Oct 30 21:21:27.784074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.782696    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"]
Oct 30 21:21:27.784074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:27.782943    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy" containerID="cri-o://7ec4297f203589c7d2e3ce1009bc4fd9133c61e1cb7dc159151996a76ed55f55" gracePeriod=30
Oct 30 21:21:28.052645 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.043638    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"]
Oct 30 21:21:28.100093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.100038    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"]
Oct 30 21:21:28.345626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.345544    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"]
Oct 30 21:21:28.353778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.353742    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"]
Oct 30 21:21:28.353935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.353896    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:28.358037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.358000    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:21:28.358300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.358273    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:21:28.385916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.385892    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:21:28.386047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.385984    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:21:28.406949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.406914    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x\" (UID: \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:28.407157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.407139    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d5wh\" (UniqueName: \"kubernetes.io/projected/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-kube-api-access-7d5wh\") pod \"maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x\" (UID: \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:28.407286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.407269    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x\" (UID: \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:28.534551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.534508    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d5wh\" (UniqueName: \"kubernetes.io/projected/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-kube-api-access-7d5wh\") pod \"maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x\" (UID: \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:28.534906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.534887    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x\" (UID: \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:28.535122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.535106    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x\" (UID: \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:28.536051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.536027    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x\" (UID: \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:28.551117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.551089    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x\" (UID: \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:28.552999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.552920    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d5wh\" (UniqueName: \"kubernetes.io/projected/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-kube-api-access-7d5wh\") pod \"maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x\" (UID: \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:28.672774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.672708    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:28.873824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.873782    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69" event={"ID":"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6","Type":"ContainerStarted","Data":"d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2"}
Oct 30 21:21:28.874023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.873846    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69" event={"ID":"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6","Type":"ContainerStarted","Data":"3110fb4680dc583924431d5dd490233250bd1c8eda90d96c56e72a5f5a686633"}
Oct 30 21:21:28.874023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.873873    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"
Oct 30 21:21:28.875528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.875498    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2" event={"ID":"27df2c10-4316-489e-9a7a-19417b131519","Type":"ContainerStarted","Data":"85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9"}
Oct 30 21:21:28.875669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.875543    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2" event={"ID":"27df2c10-4316-489e-9a7a-19417b131519","Type":"ContainerStarted","Data":"257550a2a0e3794b2f9e70020564cc5ed0ff14c9c2821625b631fb7c5c74188a"}
Oct 30 21:21:28.875669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.875603    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"
Oct 30 21:21:28.892564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.892524    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69" podStartSLOduration=2.892511823 podStartE2EDuration="2.892511823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:28.891057499 +0000 UTC m=+1201.937616654" watchObservedRunningTime="2026-10-30 21:21:28.892511823 +0000 UTC m=+1201.939070968"
Oct 30 21:21:28.913083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.913034    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2" podStartSLOduration=2.9130217910000002 podStartE2EDuration="2.913021791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:28.910090217 +0000 UTC m=+1201.956649362" watchObservedRunningTime="2026-10-30 21:21:28.913021791 +0000 UTC m=+1201.959580967"
Oct 30 21:21:28.962657 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:21:28.962629    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2d9c6a_980d_430c_a197_b94bcdb4f5ac.slice/crio-9a763e6dd86e91c00f6aea856e88c40d1d5e116b0f845d7e9c65362d8c982910 WatchSource:0}: Error finding container 9a763e6dd86e91c00f6aea856e88c40d1d5e116b0f845d7e9c65362d8c982910: Status 404 returned error can't find the container with id 9a763e6dd86e91c00f6aea856e88c40d1d5e116b0f845d7e9c65362d8c982910
Oct 30 21:21:28.962772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:28.962711    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"]
Oct 30 21:21:29.081675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:29.081637    4918 scope.go:122] "RemoveContainer" containerID="dba9860a377cd7c3cb771be62a6db4bfb6241a75678890c55aac4d49289b778a"
Oct 30 21:21:29.117616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:29.117593    4918 scope.go:122] "RemoveContainer" containerID="e0dbc1e5c2ea3378b127061d625d1e356c8dd4b83f9c3cc7f7da128e7754ebb2"
Oct 30 21:21:29.144087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:29.144065    4918 scope.go:122] "RemoveContainer" containerID="8851ae03b618000b24d6082a71096a74615f2ca72f54d47f3e8248c76a3a1b38"
Oct 30 21:21:29.160085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:29.160061    4918 scope.go:122] "RemoveContainer" containerID="a96893ad7dc069f54f4452f2ea32988088005ed01b7eb90fc048b7f219611855"
Oct 30 21:21:29.171847 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:21:29.171813    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:21:29.173673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:29.173656    4918 scope.go:122] "RemoveContainer" containerID="89ea5ea799292da486df529a8c0099d32604c86b5d1c83998c14cee6d4e76de0"
Oct 30 21:21:29.926939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:29.925351    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x" event={"ID":"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac","Type":"ContainerStarted","Data":"72f468a25db88c736d167464dd8913dfcbcff3427d07e2cc28ec8008f8f6ca02"}
Oct 30 21:21:29.926939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:29.925485    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x" event={"ID":"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac","Type":"ContainerStarted","Data":"9a763e6dd86e91c00f6aea856e88c40d1d5e116b0f845d7e9c65362d8c982910"}
Oct 30 21:21:29.957548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:29.957502    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x" podStartSLOduration=1.957488357 podStartE2EDuration="1.957488357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:29.952258128 +0000 UTC m=+1202.998817284" watchObservedRunningTime="2026-10-30 21:21:29.957488357 +0000 UTC m=+1203.004047501"
Oct 30 21:21:30.038314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.038286    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:21:30.042015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.041993    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:21:30.078246 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.078219    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"]
Oct 30 21:21:30.082589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.082573    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"
Oct 30 21:21:30.091832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.091807    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"]
Oct 30 21:21:30.419735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.379127    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-09eb50-dockercfg-lpr2x\""
Oct 30 21:21:30.430526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.430487    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-09eb50\""
Oct 30 21:21:30.442485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.442451    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjxn\" (UniqueName: \"kubernetes.io/projected/ae67cf62-b689-4557-aa4b-4e0d975e75ef-kube-api-access-vwjxn\") pod \"payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c\" (UID: \"ae67cf62-b689-4557-aa4b-4e0d975e75ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"
Oct 30 21:21:30.442709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.442548    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae67cf62-b689-4557-aa4b-4e0d975e75ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c\" (UID: \"ae67cf62-b689-4557-aa4b-4e0d975e75ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"
Oct 30 21:21:30.544496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.544456    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjxn\" (UniqueName: \"kubernetes.io/projected/ae67cf62-b689-4557-aa4b-4e0d975e75ef-kube-api-access-vwjxn\") pod \"payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c\" (UID: \"ae67cf62-b689-4557-aa4b-4e0d975e75ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"
Oct 30 21:21:30.544663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.544535    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae67cf62-b689-4557-aa4b-4e0d975e75ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c\" (UID: \"ae67cf62-b689-4557-aa4b-4e0d975e75ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"
Oct 30 21:21:30.545186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.545166    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae67cf62-b689-4557-aa4b-4e0d975e75ef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c\" (UID: \"ae67cf62-b689-4557-aa4b-4e0d975e75ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"
Oct 30 21:21:30.553819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.553796    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjxn\" (UniqueName: \"kubernetes.io/projected/ae67cf62-b689-4557-aa4b-4e0d975e75ef-kube-api-access-vwjxn\") pod \"payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c\" (UID: \"ae67cf62-b689-4557-aa4b-4e0d975e75ef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"
Oct 30 21:21:30.698702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.698577    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"
Oct 30 21:21:30.936245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.936205    4918 generic.go:356] "Generic (PLEG): container finished" podID="7f2d9c6a-980d-430c-a197-b94bcdb4f5ac" containerID="72f468a25db88c736d167464dd8913dfcbcff3427d07e2cc28ec8008f8f6ca02" exitCode=0
Oct 30 21:21:30.936439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.936312    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x" event={"ID":"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac","Type":"ContainerDied","Data":"72f468a25db88c736d167464dd8913dfcbcff3427d07e2cc28ec8008f8f6ca02"}
Oct 30 21:21:30.982654 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:21:30.982612    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae67cf62_b689_4557_aa4b_4e0d975e75ef.slice/crio-cb2cbe74f9f7bd863cb785e7431472abc5a324251286fba75e5f20dbfa99b820 WatchSource:0}: Error finding container cb2cbe74f9f7bd863cb785e7431472abc5a324251286fba75e5f20dbfa99b820: Status 404 returned error can't find the container with id cb2cbe74f9f7bd863cb785e7431472abc5a324251286fba75e5f20dbfa99b820
Oct 30 21:21:30.983271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:30.983241    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"]
Oct 30 21:21:31.024131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:31.024085    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"]
Oct 30 21:21:31.028434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:31.028411    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a019c653-qgggf"]
Oct 30 21:21:31.079222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:31.079183    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9d8f7e-082b-438a-b44f-5020dfdd616a" path="/var/lib/kubelet/pods/4b9d8f7e-082b-438a-b44f-5020dfdd616a/volumes"
Oct 30 21:21:31.946240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:31.946207    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c" event={"ID":"ae67cf62-b689-4557-aa4b-4e0d975e75ef","Type":"ContainerStarted","Data":"03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18"}
Oct 30 21:21:31.946764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:31.946261    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c" event={"ID":"ae67cf62-b689-4557-aa4b-4e0d975e75ef","Type":"ContainerStarted","Data":"cb2cbe74f9f7bd863cb785e7431472abc5a324251286fba75e5f20dbfa99b820"}
Oct 30 21:21:31.946764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:31.946289    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"
Oct 30 21:21:31.966384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:31.966325    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c" podStartSLOduration=1.966305632 podStartE2EDuration="1.966305632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:31.964229397 +0000 UTC m=+1205.010788543" watchObservedRunningTime="2026-10-30 21:21:31.966305632 +0000 UTC m=+1205.012864775"
Oct 30 21:21:32.249950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.249927    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:32.299771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.299741    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-openshift-ca\") pod \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\" (UID: \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\") "
Oct 30 21:21:32.299953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.299939    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-kube-api-access-7d5wh\" (UniqueName: \"kubernetes.io/projected/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-kube-api-access-7d5wh\") pod \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\" (UID: \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\") "
Oct 30 21:21:32.300000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.299965    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-xks-ca\") pod \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\" (UID: \"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac\") "
Oct 30 21:21:32.300085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.300064    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-openshift-ca" pod "7f2d9c6a-980d-430c-a197-b94bcdb4f5ac" (UID: "7f2d9c6a-980d-430c-a197-b94bcdb4f5ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:32.300682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.300659    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:32.302237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.302210    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-xks-ca" pod "7f2d9c6a-980d-430c-a197-b94bcdb4f5ac" (UID: "7f2d9c6a-980d-430c-a197-b94bcdb4f5ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:21:32.302381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.302359    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-kube-api-access-7d5wh" pod "7f2d9c6a-980d-430c-a197-b94bcdb4f5ac" (UID: "7f2d9c6a-980d-430c-a197-b94bcdb4f5ac"). InnerVolumeSpecName "kube-api-access-7d5wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:32.402231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.402198    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d5wh\" (UniqueName: \"kubernetes.io/projected/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-kube-api-access-7d5wh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:32.402231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.402227    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:32.956097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.956068    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"
Oct 30 21:21:32.956512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.956126    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x" event={"ID":"7f2d9c6a-980d-430c-a197-b94bcdb4f5ac","Type":"ContainerDied","Data":"9a763e6dd86e91c00f6aea856e88c40d1d5e116b0f845d7e9c65362d8c982910"}
Oct 30 21:21:32.956512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.956176    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a763e6dd86e91c00f6aea856e88c40d1d5e116b0f845d7e9c65362d8c982910"
Oct 30 21:21:32.958589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.958562    4918 generic.go:356] "Generic (PLEG): container finished" podID="97a16952-efea-4b33-a112-c4b533c353c9" containerID="7ec4297f203589c7d2e3ce1009bc4fd9133c61e1cb7dc159151996a76ed55f55" exitCode=0
Oct 30 21:21:32.958697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.958628    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" event={"ID":"97a16952-efea-4b33-a112-c4b533c353c9","Type":"ContainerDied","Data":"7ec4297f203589c7d2e3ce1009bc4fd9133c61e1cb7dc159151996a76ed55f55"}
Oct 30 21:21:32.958697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:32.958684    4918 scope.go:122] "RemoveContainer" containerID="a51a86fc68a8f6d509a9a9aea97fb6a579cb2f0c2b998464340ccd3461089916"
Oct 30 21:21:33.075558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.075536    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:21:33.232572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.232546    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-data\") pod \"97a16952-efea-4b33-a112-c4b533c353c9\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") "
Oct 30 21:21:33.232723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.232648    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-credential-socket\") pod \"97a16952-efea-4b33-a112-c4b533c353c9\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") "
Oct 30 21:21:33.232723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.232674    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-istio-token\" (UniqueName: \"kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-istio-token\") pod \"97a16952-efea-4b33-a112-c4b533c353c9\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") "
Oct 30 21:21:33.232723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.232700    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a16952-efea-4b33-a112-c4b533c353c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a16952-efea-4b33-a112-c4b533c353c9-istiod-ca-cert\") pod \"97a16952-efea-4b33-a112-c4b533c353c9\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") "
Oct 30 21:21:33.232861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.232750    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-socket\") pod \"97a16952-efea-4b33-a112-c4b533c353c9\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") "
Oct 30 21:21:33.232861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.232785    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-kube-api-access-fbs8q\" (UniqueName: \"kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-kube-api-access-fbs8q\") pod \"97a16952-efea-4b33-a112-c4b533c353c9\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") "
Oct 30 21:21:33.232861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.232795    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-data" pod "97a16952-efea-4b33-a112-c4b533c353c9" (UID: "97a16952-efea-4b33-a112-c4b533c353c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.233019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.232923    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-envoy\") pod \"97a16952-efea-4b33-a112-c4b533c353c9\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") "
Oct 30 21:21:33.233019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.232937    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-credential-socket" pod "97a16952-efea-4b33-a112-c4b533c353c9" (UID: "97a16952-efea-4b33-a112-c4b533c353c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.233462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.233152    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97a16952-efea-4b33-a112-c4b533c353c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a16952-efea-4b33-a112-c4b533c353c9-istio-podinfo\") pod \"97a16952-efea-4b33-a112-c4b533c353c9\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") "
Oct 30 21:21:33.233462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.233151    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-socket" pod "97a16952-efea-4b33-a112-c4b533c353c9" (UID: "97a16952-efea-4b33-a112-c4b533c353c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.233462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.233193    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-certs\") pod \"97a16952-efea-4b33-a112-c4b533c353c9\" (UID: \"97a16952-efea-4b33-a112-c4b533c353c9\") "
Oct 30 21:21:33.233462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.233237    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a16952-efea-4b33-a112-c4b533c353c9-istiod-ca-cert" pod "97a16952-efea-4b33-a112-c4b533c353c9" (UID: "97a16952-efea-4b33-a112-c4b533c353c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:33.234252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.234120    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.234252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.234146    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.234252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.234164    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a16952-efea-4b33-a112-c4b533c353c9-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.234252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.234181    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.234739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.234709    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-certs" pod "97a16952-efea-4b33-a112-c4b533c353c9" (UID: "97a16952-efea-4b33-a112-c4b533c353c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.235822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.235800    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-kube-api-access-fbs8q" pod "97a16952-efea-4b33-a112-c4b533c353c9" (UID: "97a16952-efea-4b33-a112-c4b533c353c9"). InnerVolumeSpecName "kube-api-access-fbs8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:33.236081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.236055    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-istio-token" pod "97a16952-efea-4b33-a112-c4b533c353c9" (UID: "97a16952-efea-4b33-a112-c4b533c353c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:33.236200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.236097    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97a16952-efea-4b33-a112-c4b533c353c9-istio-podinfo" pod "97a16952-efea-4b33-a112-c4b533c353c9" (UID: "97a16952-efea-4b33-a112-c4b533c353c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:21:33.236801 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.236771    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-envoy" pod "97a16952-efea-4b33-a112-c4b533c353c9" (UID: "97a16952-efea-4b33-a112-c4b533c353c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:33.335211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.335131    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.335211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.335160    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a16952-efea-4b33-a112-c4b533c353c9-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.335211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.335169    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a16952-efea-4b33-a112-c4b533c353c9-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.335211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.335179    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.335211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.335188    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbs8q\" (UniqueName: \"kubernetes.io/projected/97a16952-efea-4b33-a112-c4b533c353c9-kube-api-access-fbs8q\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:33.967745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.967721    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"
Oct 30 21:21:33.968212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.967750    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt" event={"ID":"97a16952-efea-4b33-a112-c4b533c353c9","Type":"ContainerDied","Data":"65f293baec6c13ceb15503226b31921465ffa9aa72bee790d347a9e3179459c8"}
Oct 30 21:21:33.968212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.967828    4918 scope.go:122] "RemoveContainer" containerID="7ec4297f203589c7d2e3ce1009bc4fd9133c61e1cb7dc159151996a76ed55f55"
Oct 30 21:21:33.996824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.995068    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"]
Oct 30 21:21:33.999213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:33.999192    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3ac6bf-openshift-default-c786786b9-8w7wt"]
Oct 30 21:21:34.030240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:34.030215    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"]
Oct 30 21:21:34.039523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:34.039496    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5da633f5-trnrj"]
Oct 30 21:21:35.027312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:35.027272    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"]
Oct 30 21:21:35.031082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:35.031060    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e37648a2-6jwhl"]
Oct 30 21:21:35.079939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:35.079899    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc720ed-be58-4f75-a5d2-5599b7ab7f4f" path="/var/lib/kubelet/pods/7bc720ed-be58-4f75-a5d2-5599b7ab7f4f/volumes"
Oct 30 21:21:35.080505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:35.080485    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c8b946a-fa86-416f-8c00-303fff251ca3" path="/var/lib/kubelet/pods/7c8b946a-fa86-416f-8c00-303fff251ca3/volumes"
Oct 30 21:21:35.081025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:35.081006    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a16952-efea-4b33-a112-c4b533c353c9" path="/var/lib/kubelet/pods/97a16952-efea-4b33-a112-c4b533c353c9/volumes"
Oct 30 21:21:37.023096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:37.023007    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"]
Oct 30 21:21:37.026971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:37.026945    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5779dc1e-vlncb"]
Oct 30 21:21:37.078716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:37.078687    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69dd6c25-8430-4b8e-97a7-dba7627858b5" path="/var/lib/kubelet/pods/69dd6c25-8430-4b8e-97a7-dba7627858b5/volumes"
Oct 30 21:21:39.928533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:39.928504    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"
Oct 30 21:21:39.928965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:39.928578    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"
Oct 30 21:21:43.011426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:43.010600    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"
Oct 30 21:21:44.361738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:44.361700    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"]
Oct 30 21:21:44.362223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:44.361989    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v" podUID="9dc77c6a-4698-4ab3-bb08-d44b211720c8" containerName="payload-pre-processing" containerID="cri-o://4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91" gracePeriod=30
Oct 30 21:21:44.918046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:44.918020    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"]
Oct 30 21:21:44.927985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:44.926998    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-60c26c-6mc2x"]
Oct 30 21:21:44.969782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:44.969760    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"
Oct 30 21:21:45.127481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.112816    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc77c6a-4698-4ab3-bb08-d44b211720c8-kube-api-access-bffqt\" (UniqueName: \"kubernetes.io/projected/9dc77c6a-4698-4ab3-bb08-d44b211720c8-kube-api-access-bffqt\") pod \"9dc77c6a-4698-4ab3-bb08-d44b211720c8\" (UID: \"9dc77c6a-4698-4ab3-bb08-d44b211720c8\") "
Oct 30 21:21:45.147953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.146846    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc77c6a-4698-4ab3-bb08-d44b211720c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc77c6a-4698-4ab3-bb08-d44b211720c8-plugins-config-volume\") pod \"9dc77c6a-4698-4ab3-bb08-d44b211720c8\" (UID: \"9dc77c6a-4698-4ab3-bb08-d44b211720c8\") "
Oct 30 21:21:45.147953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.147423    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc77c6a-4698-4ab3-bb08-d44b211720c8-plugins-config-volume" pod "9dc77c6a-4698-4ab3-bb08-d44b211720c8" (UID: "9dc77c6a-4698-4ab3-bb08-d44b211720c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:45.149107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.149090    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9dc77c6a-4698-4ab3-bb08-d44b211720c8-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:45.149280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.149259    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2d9c6a-980d-430c-a197-b94bcdb4f5ac" path="/var/lib/kubelet/pods/7f2d9c6a-980d-430c-a197-b94bcdb4f5ac/volumes"
Oct 30 21:21:45.149603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.149586    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc77c6a-4698-4ab3-bb08-d44b211720c8-kube-api-access-bffqt" pod "9dc77c6a-4698-4ab3-bb08-d44b211720c8" (UID: "9dc77c6a-4698-4ab3-bb08-d44b211720c8"). InnerVolumeSpecName "kube-api-access-bffqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:45.149716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.149649    4918 generic.go:356] "Generic (PLEG): container finished" podID="9dc77c6a-4698-4ab3-bb08-d44b211720c8" containerID="4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91" exitCode=0
Oct 30 21:21:45.149894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.149716    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"
Oct 30 21:21:45.152028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.150501    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v" event={"ID":"9dc77c6a-4698-4ab3-bb08-d44b211720c8","Type":"ContainerDied","Data":"4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91"}
Oct 30 21:21:45.152028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.150553    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v" event={"ID":"9dc77c6a-4698-4ab3-bb08-d44b211720c8","Type":"ContainerDied","Data":"431a83738869a7469345ea6402c3fef088bf75a0392bd3d2c56d2fcf5f5e24e1"}
Oct 30 21:21:45.152028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.150585    4918 scope.go:122] "RemoveContainer" containerID="4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91"
Oct 30 21:21:45.171332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.171318    4918 scope.go:122] "RemoveContainer" containerID="4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91"
Oct 30 21:21:45.171639 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:21:45.171622    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91\": container with ID starting with 4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91 not found: ID does not exist" containerID="4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91"
Oct 30 21:21:45.171702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.171649    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91"} err="failed to get container status \"4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91\": rpc error: code = NotFound desc = could not find container \"4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91\": container with ID starting with 4cba02accc2bd454c0d5646c07b2652acd2ee3621fb0153825095cbb8e3f6f91 not found: ID does not exist"
Oct 30 21:21:45.185142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.185122    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"]
Oct 30 21:21:45.189690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.189670    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60c26c-8468cb5b79-r778v"]
Oct 30 21:21:45.250540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:45.250510    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bffqt\" (UniqueName: \"kubernetes.io/projected/9dc77c6a-4698-4ab3-bb08-d44b211720c8-kube-api-access-bffqt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:47.078516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:47.078487    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc77c6a-4698-4ab3-bb08-d44b211720c8" path="/var/lib/kubelet/pods/9dc77c6a-4698-4ab3-bb08-d44b211720c8/volumes"
Oct 30 21:21:49.052529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:49.052488    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"]
Oct 30 21:21:49.057328 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:49.057303    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4faa5985-5nzkj"]
Oct 30 21:21:49.080141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:49.080104    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6dac54-3b54-403c-991c-4dcc528478f6" path="/var/lib/kubelet/pods/9a6dac54-3b54-403c-991c-4dcc528478f6/volumes"
Oct 30 21:21:51.850324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.850290    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"]
Oct 30 21:21:51.850975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.850961    4918 cpu_manager.go:414] "containerMap: removing container" podUID="7f2d9c6a-980d-430c-a197-b94bcdb4f5ac" containerName="revoke-keys"
Oct 30 21:21:51.851031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.850977    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2d9c6a-980d-430c-a197-b94bcdb4f5ac" containerName="revoke-keys"
Oct 30 21:21:51.851031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.850989    4918 cpu_manager.go:414] "containerMap: removing container" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy"
Oct 30 21:21:51.851031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.850994    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy"
Oct 30 21:21:51.851031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.851026    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9dc77c6a-4698-4ab3-bb08-d44b211720c8" containerName="payload-pre-processing"
Oct 30 21:21:51.851031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.851031    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc77c6a-4698-4ab3-bb08-d44b211720c8" containerName="payload-pre-processing"
Oct 30 21:21:51.851201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.851116    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2d9c6a-980d-430c-a197-b94bcdb4f5ac" containerName="revoke-keys"
Oct 30 21:21:51.851201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.851129    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy"
Oct 30 21:21:51.851201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.851139    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy"
Oct 30 21:21:51.851201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.851157    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc77c6a-4698-4ab3-bb08-d44b211720c8" containerName="payload-pre-processing"
Oct 30 21:21:51.856347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.856328    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:51.862078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.862062    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:21:51.862179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.862167    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:21:51.865076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.865053    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:21:51.866536 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.866511    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:21:51.869909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.869888    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"]
Oct 30 21:21:51.903248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.903224    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx4h7\" (UniqueName: \"kubernetes.io/projected/d85192bf-e597-4206-8752-6f886fc08ffa-kube-api-access-fx4h7\") pod \"maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b\" (UID: \"d85192bf-e597-4206-8752-6f886fc08ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:51.903439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.903423    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b\" (UID: \"d85192bf-e597-4206-8752-6f886fc08ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:51.903516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:51.903500    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b\" (UID: \"d85192bf-e597-4206-8752-6f886fc08ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:52.016416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:52.007323    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b\" (UID: \"d85192bf-e597-4206-8752-6f886fc08ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:52.016858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:52.016822    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b\" (UID: \"d85192bf-e597-4206-8752-6f886fc08ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:52.030710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:52.030669    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx4h7\" (UniqueName: \"kubernetes.io/projected/d85192bf-e597-4206-8752-6f886fc08ffa-kube-api-access-fx4h7\") pod \"maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b\" (UID: \"d85192bf-e597-4206-8752-6f886fc08ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:52.031516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:52.012610    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b\" (UID: \"d85192bf-e597-4206-8752-6f886fc08ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:52.043052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:52.036944    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b\" (UID: \"d85192bf-e597-4206-8752-6f886fc08ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:52.054451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:52.051485    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx4h7\" (UniqueName: \"kubernetes.io/projected/d85192bf-e597-4206-8752-6f886fc08ffa-kube-api-access-fx4h7\") pod \"maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b\" (UID: \"d85192bf-e597-4206-8752-6f886fc08ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:52.171597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:52.171529    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:52.490111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:52.490066    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"]
Oct 30 21:21:52.491016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:52.490959    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy" containerID="cri-o://9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0" gracePeriod=30
Oct 30 21:21:52.525484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:52.525463    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"]
Oct 30 21:21:52.526265 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:21:52.526239    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85192bf_e597_4206_8752_6f886fc08ffa.slice/crio-ffd096d91e898cdb50488a16f59d92d09b29c34314d98f42dbf81ffc806f656c WatchSource:0}: Error finding container ffd096d91e898cdb50488a16f59d92d09b29c34314d98f42dbf81ffc806f656c: Status 404 returned error can't find the container with id ffd096d91e898cdb50488a16f59d92d09b29c34314d98f42dbf81ffc806f656c
Oct 30 21:21:53.306464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:53.306429    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b" event={"ID":"d85192bf-e597-4206-8752-6f886fc08ffa","Type":"ContainerStarted","Data":"98bc43ff2044a5ec0f9f702e4175bb1bc1f0d41a5c870b23392107a5647ada54"}
Oct 30 21:21:53.306871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:53.306486    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b" event={"ID":"d85192bf-e597-4206-8752-6f886fc08ffa","Type":"ContainerStarted","Data":"ffd096d91e898cdb50488a16f59d92d09b29c34314d98f42dbf81ffc806f656c"}
Oct 30 21:21:53.325254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:53.325211    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b" podStartSLOduration=2.325196687 podStartE2EDuration="2.325196687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:53.32193491 +0000 UTC m=+1226.368494055" watchObservedRunningTime="2026-10-30 21:21:53.325196687 +0000 UTC m=+1226.371755832"
Oct 30 21:21:53.929727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:53.929692    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"]
Oct 30 21:21:54.316545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:54.316506    4918 generic.go:356] "Generic (PLEG): container finished" podID="d85192bf-e597-4206-8752-6f886fc08ffa" containerID="98bc43ff2044a5ec0f9f702e4175bb1bc1f0d41a5c870b23392107a5647ada54" exitCode=0
Oct 30 21:21:54.316928 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:54.316596    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b" event={"ID":"d85192bf-e597-4206-8752-6f886fc08ffa","Type":"ContainerDied","Data":"98bc43ff2044a5ec0f9f702e4175bb1bc1f0d41a5c870b23392107a5647ada54"}
Oct 30 21:21:55.346254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.346223    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"]
Oct 30 21:21:55.347034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.347015    4918 cpu_manager.go:414] "containerMap: removing container" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy"
Oct 30 21:21:55.347099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.347036    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a16952-efea-4b33-a112-c4b533c353c9" containerName="istio-proxy"
Oct 30 21:21:55.350838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.350819    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"
Oct 30 21:21:55.355857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.355834    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-6563b6\""
Oct 30 21:21:55.363540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.360454    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"]
Oct 30 21:21:55.365713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.365689    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"
Oct 30 21:21:55.370683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.370641    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"]
Oct 30 21:21:55.377564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.377490    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"]
Oct 30 21:21:55.378508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.378484    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-6563b6-dockercfg-6w5bv\""
Oct 30 21:21:55.424531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.424498    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdt5f\" (UniqueName: \"kubernetes.io/projected/df366a2f-ee2a-44c4-9b91-1d89db6d1475-kube-api-access-vdt5f\") pod \"payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm\" (UID: \"df366a2f-ee2a-44c4-9b91-1d89db6d1475\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"
Oct 30 21:21:55.424776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.424754    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d95733-0e8d-4375-a11d-eec651bb170f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj\" (UID: \"f8d95733-0e8d-4375-a11d-eec651bb170f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"
Oct 30 21:21:55.424902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.424884    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df366a2f-ee2a-44c4-9b91-1d89db6d1475-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm\" (UID: \"df366a2f-ee2a-44c4-9b91-1d89db6d1475\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"
Oct 30 21:21:55.425276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.424998    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqw92\" (UniqueName: \"kubernetes.io/projected/f8d95733-0e8d-4375-a11d-eec651bb170f-kube-api-access-kqw92\") pod \"payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj\" (UID: \"f8d95733-0e8d-4375-a11d-eec651bb170f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"
Oct 30 21:21:55.544427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.541604    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdt5f\" (UniqueName: \"kubernetes.io/projected/df366a2f-ee2a-44c4-9b91-1d89db6d1475-kube-api-access-vdt5f\") pod \"payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm\" (UID: \"df366a2f-ee2a-44c4-9b91-1d89db6d1475\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"
Oct 30 21:21:55.544427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.541665    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d95733-0e8d-4375-a11d-eec651bb170f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj\" (UID: \"f8d95733-0e8d-4375-a11d-eec651bb170f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"
Oct 30 21:21:55.544427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.541700    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df366a2f-ee2a-44c4-9b91-1d89db6d1475-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm\" (UID: \"df366a2f-ee2a-44c4-9b91-1d89db6d1475\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"
Oct 30 21:21:55.544427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.541805    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqw92\" (UniqueName: \"kubernetes.io/projected/f8d95733-0e8d-4375-a11d-eec651bb170f-kube-api-access-kqw92\") pod \"payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj\" (UID: \"f8d95733-0e8d-4375-a11d-eec651bb170f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"
Oct 30 21:21:55.564440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.545600    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d95733-0e8d-4375-a11d-eec651bb170f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj\" (UID: \"f8d95733-0e8d-4375-a11d-eec651bb170f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"
Oct 30 21:21:55.564440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.546360    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df366a2f-ee2a-44c4-9b91-1d89db6d1475-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm\" (UID: \"df366a2f-ee2a-44c4-9b91-1d89db6d1475\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"
Oct 30 21:21:55.742411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.735168    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqw92\" (UniqueName: \"kubernetes.io/projected/f8d95733-0e8d-4375-a11d-eec651bb170f-kube-api-access-kqw92\") pod \"payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj\" (UID: \"f8d95733-0e8d-4375-a11d-eec651bb170f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"
Oct 30 21:21:55.742411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.735878    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdt5f\" (UniqueName: \"kubernetes.io/projected/df366a2f-ee2a-44c4-9b91-1d89db6d1475-kube-api-access-vdt5f\") pod \"payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm\" (UID: \"df366a2f-ee2a-44c4-9b91-1d89db6d1475\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"
Oct 30 21:21:55.869439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.857960    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:55.957314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.957284    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85192bf-e597-4206-8752-6f886fc08ffa-kube-api-access-fx4h7\" (UniqueName: \"kubernetes.io/projected/d85192bf-e597-4206-8752-6f886fc08ffa-kube-api-access-fx4h7\") pod \"d85192bf-e597-4206-8752-6f886fc08ffa\" (UID: \"d85192bf-e597-4206-8752-6f886fc08ffa\") "
Oct 30 21:21:55.957532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.957374    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-openshift-ca\") pod \"d85192bf-e597-4206-8752-6f886fc08ffa\" (UID: \"d85192bf-e597-4206-8752-6f886fc08ffa\") "
Oct 30 21:21:55.957532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.957444    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-xks-ca\") pod \"d85192bf-e597-4206-8752-6f886fc08ffa\" (UID: \"d85192bf-e597-4206-8752-6f886fc08ffa\") "
Oct 30 21:21:55.957762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.957737    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-openshift-ca" pod "d85192bf-e597-4206-8752-6f886fc08ffa" (UID: "d85192bf-e597-4206-8752-6f886fc08ffa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:55.957996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.957981    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:55.959625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.959605    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-xks-ca" pod "d85192bf-e597-4206-8752-6f886fc08ffa" (UID: "d85192bf-e597-4206-8752-6f886fc08ffa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:21:55.959695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.959628    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85192bf-e597-4206-8752-6f886fc08ffa-kube-api-access-fx4h7" pod "d85192bf-e597-4206-8752-6f886fc08ffa" (UID: "d85192bf-e597-4206-8752-6f886fc08ffa"). InnerVolumeSpecName "kube-api-access-fx4h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:56.004880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:55.970513    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"
Oct 30 21:21:56.004880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:56.002868    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"
Oct 30 21:21:56.101416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:56.087799    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85192bf-e597-4206-8752-6f886fc08ffa-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:56.102325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:56.102300    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx4h7\" (UniqueName: \"kubernetes.io/projected/d85192bf-e597-4206-8752-6f886fc08ffa-kube-api-access-fx4h7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:56.338299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:56.338267    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b" event={"ID":"d85192bf-e597-4206-8752-6f886fc08ffa","Type":"ContainerDied","Data":"ffd096d91e898cdb50488a16f59d92d09b29c34314d98f42dbf81ffc806f656c"}
Oct 30 21:21:56.338511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:56.338307    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"
Oct 30 21:21:56.338511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:56.338318    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffd096d91e898cdb50488a16f59d92d09b29c34314d98f42dbf81ffc806f656c"
Oct 30 21:21:56.482170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:56.482134    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"]
Oct 30 21:21:56.482628 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:21:56.482512    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf366a2f_ee2a_44c4_9b91_1d89db6d1475.slice/crio-7dc709f04f392e96dfb2ce4fe9d3d17c8ca032c16f0d4484b82febb9f43cd5dd WatchSource:0}: Error finding container 7dc709f04f392e96dfb2ce4fe9d3d17c8ca032c16f0d4484b82febb9f43cd5dd: Status 404 returned error can't find the container with id 7dc709f04f392e96dfb2ce4fe9d3d17c8ca032c16f0d4484b82febb9f43cd5dd
Oct 30 21:21:56.498191 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:21:56.498004    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d95733_0e8d_4375_a11d_eec651bb170f.slice/crio-278906efec59c89d80b5760b90c1b867dbf043e1ed5aff9a66f1a8185956fa08 WatchSource:0}: Error finding container 278906efec59c89d80b5760b90c1b867dbf043e1ed5aff9a66f1a8185956fa08: Status 404 returned error can't find the container with id 278906efec59c89d80b5760b90c1b867dbf043e1ed5aff9a66f1a8185956fa08
Oct 30 21:21:56.498288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:56.498211    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"]
Oct 30 21:21:57.349745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.349715    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj" event={"ID":"f8d95733-0e8d-4375-a11d-eec651bb170f","Type":"ContainerStarted","Data":"5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86"}
Oct 30 21:21:57.349998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.349769    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj" event={"ID":"f8d95733-0e8d-4375-a11d-eec651bb170f","Type":"ContainerStarted","Data":"278906efec59c89d80b5760b90c1b867dbf043e1ed5aff9a66f1a8185956fa08"}
Oct 30 21:21:57.350066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.350042    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"
Oct 30 21:21:57.351625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.351599    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm" event={"ID":"df366a2f-ee2a-44c4-9b91-1d89db6d1475","Type":"ContainerStarted","Data":"424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d"}
Oct 30 21:21:57.351768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.351647    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm" event={"ID":"df366a2f-ee2a-44c4-9b91-1d89db6d1475","Type":"ContainerStarted","Data":"7dc709f04f392e96dfb2ce4fe9d3d17c8ca032c16f0d4484b82febb9f43cd5dd"}
Oct 30 21:21:57.351768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.351682    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"
Oct 30 21:21:57.412377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.412136    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj" podStartSLOduration=2.4121101 podStartE2EDuration="2.4121101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:57.384843269 +0000 UTC m=+1230.431402416" watchObservedRunningTime="2026-10-30 21:21:57.4121101 +0000 UTC m=+1230.458669245"
Oct 30 21:21:57.830619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.830592    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:21:57.856414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.856344    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm" podStartSLOduration=2.856330017 podStartE2EDuration="2.856330017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:57.425044293 +0000 UTC m=+1230.471603460" watchObservedRunningTime="2026-10-30 21:21:57.856330017 +0000 UTC m=+1230.902889205"
Oct 30 21:21:57.870960 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.870878    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-envoy\") pod \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") "
Oct 30 21:21:57.871115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.871039    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istiod-ca-cert\") pod \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") "
Oct 30 21:21:57.871115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.871089    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-credential-socket\") pod \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") "
Oct 30 21:21:57.871232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.871186    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-certs\") pod \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") "
Oct 30 21:21:57.871232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.871215    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-token\" (UniqueName: \"kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-token\") pod \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") "
Oct 30 21:21:57.871344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.871245    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-kube-api-access-zj4nf\" (UniqueName: \"kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-kube-api-access-zj4nf\") pod \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") "
Oct 30 21:21:57.871344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.871333    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-podinfo\") pod \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") "
Oct 30 21:21:57.871509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.871368    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-socket\") pod \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") "
Oct 30 21:21:57.871509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.871472    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-credential-socket" pod "5e0b52b8-a075-4ec6-8ee4-79fec413717d" (UID: "5e0b52b8-a075-4ec6-8ee4-79fec413717d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:57.871620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.871516    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istiod-ca-cert" pod "5e0b52b8-a075-4ec6-8ee4-79fec413717d" (UID: "5e0b52b8-a075-4ec6-8ee4-79fec413717d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:57.871620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.871523    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-data\") pod \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\" (UID: \"5e0b52b8-a075-4ec6-8ee4-79fec413717d\") "
Oct 30 21:21:57.872026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.871995    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-data" pod "5e0b52b8-a075-4ec6-8ee4-79fec413717d" (UID: "5e0b52b8-a075-4ec6-8ee4-79fec413717d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:57.872261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.872237    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-certs" pod "5e0b52b8-a075-4ec6-8ee4-79fec413717d" (UID: "5e0b52b8-a075-4ec6-8ee4-79fec413717d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:57.872463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.872439    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-socket" pod "5e0b52b8-a075-4ec6-8ee4-79fec413717d" (UID: "5e0b52b8-a075-4ec6-8ee4-79fec413717d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:57.872876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.872847    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:57.872966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.872884    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:57.872966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.872898    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:57.872966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.872911    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:57.874508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.874482    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-envoy" pod "5e0b52b8-a075-4ec6-8ee4-79fec413717d" (UID: "5e0b52b8-a075-4ec6-8ee4-79fec413717d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:21:57.875124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.875103    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-podinfo" pod "5e0b52b8-a075-4ec6-8ee4-79fec413717d" (UID: "5e0b52b8-a075-4ec6-8ee4-79fec413717d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:21:57.875248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.875224    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-kube-api-access-zj4nf" pod "5e0b52b8-a075-4ec6-8ee4-79fec413717d" (UID: "5e0b52b8-a075-4ec6-8ee4-79fec413717d"). InnerVolumeSpecName "kube-api-access-zj4nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:57.879579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.879554    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-token" pod "5e0b52b8-a075-4ec6-8ee4-79fec413717d" (UID: "5e0b52b8-a075-4ec6-8ee4-79fec413717d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:57.973871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.973832    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:57.973871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.973865    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj4nf\" (UniqueName: \"kubernetes.io/projected/5e0b52b8-a075-4ec6-8ee4-79fec413717d-kube-api-access-zj4nf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:57.974070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.973880    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:57.974070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.973892    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:57.974070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:57.973905    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e0b52b8-a075-4ec6-8ee4-79fec413717d-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:58.361278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.361248    4918 generic.go:356] "Generic (PLEG): container finished" podID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerID="9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0" exitCode=0
Oct 30 21:21:58.361466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.361298    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"
Oct 30 21:21:58.361466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.361332    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" event={"ID":"5e0b52b8-a075-4ec6-8ee4-79fec413717d","Type":"ContainerDied","Data":"9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0"}
Oct 30 21:21:58.361466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.361403    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf" event={"ID":"5e0b52b8-a075-4ec6-8ee4-79fec413717d","Type":"ContainerDied","Data":"b09741929f2287fe0a5f33610680cdf917e7672ec1eb9246a941ec5d2b9d77fb"}
Oct 30 21:21:58.361466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.361419    4918 scope.go:122] "RemoveContainer" containerID="9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0"
Oct 30 21:21:58.373914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.373899    4918 scope.go:122] "RemoveContainer" containerID="a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf"
Oct 30 21:21:58.385201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.385182    4918 scope.go:122] "RemoveContainer" containerID="9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0"
Oct 30 21:21:58.385465 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:21:58.385444    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0\": container with ID starting with 9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0 not found: ID does not exist" containerID="9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0"
Oct 30 21:21:58.385538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.385473    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0"} err="failed to get container status \"9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0\": rpc error: code = NotFound desc = could not find container \"9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0\": container with ID starting with 9cabced07afd9fd9f02538bff661fdfbef8d4c60ba0a063cf073a204165be0f0 not found: ID does not exist"
Oct 30 21:21:58.385538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.385489    4918 scope.go:122] "RemoveContainer" containerID="a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf"
Oct 30 21:21:58.385770 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:21:58.385749    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf\": container with ID starting with a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf not found: ID does not exist" containerID="a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf"
Oct 30 21:21:58.385829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.385776    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf"} err="failed to get container status \"a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf\": rpc error: code = NotFound desc = could not find container \"a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf\": container with ID starting with a881f9dca8bded449f8cb568b668d5fb3bd9a634aa914370d527c7a27a95a6cf not found: ID does not exist"
Oct 30 21:21:58.387380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.387358    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"]
Oct 30 21:21:58.393420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.393383    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-60c26c-openshift-default-7bfc889f7c-8pswf"]
Oct 30 21:21:58.930075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:58.930030    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c" podUID="ae67cf62-b689-4557-aa4b-4e0d975e75ef" containerName="payload-processing" containerID="cri-o://03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18" gracePeriod=25
Oct 30 21:21:59.080154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.080119    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" path="/var/lib/kubelet/pods/5e0b52b8-a075-4ec6-8ee4-79fec413717d/volumes"
Oct 30 21:21:59.183108 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:21:59.183020    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:21:59.236838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.236813    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"
Oct 30 21:21:59.302114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.302074    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae67cf62-b689-4557-aa4b-4e0d975e75ef-kube-api-access-vwjxn\" (UniqueName: \"kubernetes.io/projected/ae67cf62-b689-4557-aa4b-4e0d975e75ef-kube-api-access-vwjxn\") pod \"ae67cf62-b689-4557-aa4b-4e0d975e75ef\" (UID: \"ae67cf62-b689-4557-aa4b-4e0d975e75ef\") "
Oct 30 21:21:59.302311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.302290    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae67cf62-b689-4557-aa4b-4e0d975e75ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae67cf62-b689-4557-aa4b-4e0d975e75ef-plugins-config-volume\") pod \"ae67cf62-b689-4557-aa4b-4e0d975e75ef\" (UID: \"ae67cf62-b689-4557-aa4b-4e0d975e75ef\") "
Oct 30 21:21:59.302770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.302737    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae67cf62-b689-4557-aa4b-4e0d975e75ef-plugins-config-volume" pod "ae67cf62-b689-4557-aa4b-4e0d975e75ef" (UID: "ae67cf62-b689-4557-aa4b-4e0d975e75ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:21:59.302971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.302951    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae67cf62-b689-4557-aa4b-4e0d975e75ef-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:21:59.304283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.304259    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae67cf62-b689-4557-aa4b-4e0d975e75ef-kube-api-access-vwjxn" pod "ae67cf62-b689-4557-aa4b-4e0d975e75ef" (UID: "ae67cf62-b689-4557-aa4b-4e0d975e75ef"). InnerVolumeSpecName "kube-api-access-vwjxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:21:59.371597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.371566    4918 generic.go:356] "Generic (PLEG): container finished" podID="ae67cf62-b689-4557-aa4b-4e0d975e75ef" containerID="03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18" exitCode=0
Oct 30 21:21:59.371779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.371619    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"
Oct 30 21:21:59.371779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.371654    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c" event={"ID":"ae67cf62-b689-4557-aa4b-4e0d975e75ef","Type":"ContainerDied","Data":"03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18"}
Oct 30 21:21:59.371779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.371710    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c" event={"ID":"ae67cf62-b689-4557-aa4b-4e0d975e75ef","Type":"ContainerDied","Data":"cb2cbe74f9f7bd863cb785e7431472abc5a324251286fba75e5f20dbfa99b820"}
Oct 30 21:21:59.371779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.371734    4918 scope.go:122] "RemoveContainer" containerID="03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18"
Oct 30 21:21:59.383585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.383567    4918 scope.go:122] "RemoveContainer" containerID="03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18"
Oct 30 21:21:59.383841 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:21:59.383822    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18\": container with ID starting with 03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18 not found: ID does not exist" containerID="03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18"
Oct 30 21:21:59.383895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.383849    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18"} err="failed to get container status \"03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18\": rpc error: code = NotFound desc = could not find container \"03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18\": container with ID starting with 03ac3e5ba82362f3e3610234dfbb74a79bc033bc517655c8325ad51238098e18 not found: ID does not exist"
Oct 30 21:21:59.396242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.396217    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"]
Oct 30 21:21:59.402430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.402406    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09eb50-549454db65-jn48c"]
Oct 30 21:21:59.404505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:21:59.404474    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwjxn\" (UniqueName: \"kubernetes.io/projected/ae67cf62-b689-4557-aa4b-4e0d975e75ef-kube-api-access-vwjxn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:01.078796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.078760    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae67cf62-b689-4557-aa4b-4e0d975e75ef" path="/var/lib/kubelet/pods/ae67cf62-b689-4557-aa4b-4e0d975e75ef/volumes"
Oct 30 21:22:01.216680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.216650    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"]
Oct 30 21:22:01.217410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.217375    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ae67cf62-b689-4557-aa4b-4e0d975e75ef" containerName="payload-processing"
Oct 30 21:22:01.217477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.217410    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae67cf62-b689-4557-aa4b-4e0d975e75ef" containerName="payload-processing"
Oct 30 21:22:01.217477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.217441    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy"
Oct 30 21:22:01.217477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.217447    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy"
Oct 30 21:22:01.217579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.217460    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d85192bf-e597-4206-8752-6f886fc08ffa" containerName="revoke-keys"
Oct 30 21:22:01.217579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.217491    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85192bf-e597-4206-8752-6f886fc08ffa" containerName="revoke-keys"
Oct 30 21:22:01.217639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.217624    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy"
Oct 30 21:22:01.217639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.217632    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85192bf-e597-4206-8752-6f886fc08ffa" containerName="revoke-keys"
Oct 30 21:22:01.217639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.217638    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy"
Oct 30 21:22:01.217727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.217649    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae67cf62-b689-4557-aa4b-4e0d975e75ef" containerName="payload-processing"
Oct 30 21:22:01.222668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.222645    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"]
Oct 30 21:22:01.222788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.222765    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"
Oct 30 21:22:01.223335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.223305    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy"
Oct 30 21:22:01.223335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.223323    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0b52b8-a075-4ec6-8ee4-79fec413717d" containerName="istio-proxy"
Oct 30 21:22:01.226953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.226935    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"
Oct 30 21:22:01.234982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.234959    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"]
Oct 30 21:22:01.235252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.235236    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-df078c\""
Oct 30 21:22:01.235442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.235236    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-df078c-dockercfg-bcqsj\""
Oct 30 21:22:01.240674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.240654    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"]
Oct 30 21:22:01.361465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.350619    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27047a93-6fc0-4006-80a4-dbef6512129b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn\" (UID: \"27047a93-6fc0-4006-80a4-dbef6512129b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"
Oct 30 21:22:01.361465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.350904    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w\" (UID: \"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"
Oct 30 21:22:01.361465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.350941    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzhxr\" (UniqueName: \"kubernetes.io/projected/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-kube-api-access-xzhxr\") pod \"payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w\" (UID: \"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"
Oct 30 21:22:01.361465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.351007    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g72fs\" (UniqueName: \"kubernetes.io/projected/27047a93-6fc0-4006-80a4-dbef6512129b-kube-api-access-g72fs\") pod \"payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn\" (UID: \"27047a93-6fc0-4006-80a4-dbef6512129b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"
Oct 30 21:22:01.473610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.473565    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w\" (UID: \"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"
Oct 30 21:22:01.518665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.518632    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzhxr\" (UniqueName: \"kubernetes.io/projected/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-kube-api-access-xzhxr\") pod \"payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w\" (UID: \"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"
Oct 30 21:22:01.520356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.520329    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g72fs\" (UniqueName: \"kubernetes.io/projected/27047a93-6fc0-4006-80a4-dbef6512129b-kube-api-access-g72fs\") pod \"payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn\" (UID: \"27047a93-6fc0-4006-80a4-dbef6512129b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"
Oct 30 21:22:01.521453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.521429    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27047a93-6fc0-4006-80a4-dbef6512129b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn\" (UID: \"27047a93-6fc0-4006-80a4-dbef6512129b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"
Oct 30 21:22:01.523594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.521578    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w\" (UID: \"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"
Oct 30 21:22:01.523594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.522985    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27047a93-6fc0-4006-80a4-dbef6512129b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn\" (UID: \"27047a93-6fc0-4006-80a4-dbef6512129b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"
Oct 30 21:22:01.573537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.573504    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzhxr\" (UniqueName: \"kubernetes.io/projected/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-kube-api-access-xzhxr\") pod \"payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w\" (UID: \"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"
Oct 30 21:22:01.587145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.587115    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g72fs\" (UniqueName: \"kubernetes.io/projected/27047a93-6fc0-4006-80a4-dbef6512129b-kube-api-access-g72fs\") pod \"payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn\" (UID: \"27047a93-6fc0-4006-80a4-dbef6512129b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"
Oct 30 21:22:01.661227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.661126    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"]
Oct 30 21:22:01.667916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.667888    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"
Oct 30 21:22:01.676343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.676309    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"]
Oct 30 21:22:01.676846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.676824    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-09eb50-dockercfg-j4c9d\""
Oct 30 21:22:01.678609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.678592    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-09eb50\""
Oct 30 21:22:01.931412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.931285    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxlr\" (UniqueName: \"kubernetes.io/projected/536e37a1-058c-42e5-8b80-7cc2d4194711-kube-api-access-jrxlr\") pod \"payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k\" (UID: \"536e37a1-058c-42e5-8b80-7cc2d4194711\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"
Oct 30 21:22:01.931615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:01.931593    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536e37a1-058c-42e5-8b80-7cc2d4194711-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k\" (UID: \"536e37a1-058c-42e5-8b80-7cc2d4194711\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"
Oct 30 21:22:02.019153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.005334    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"
Oct 30 21:22:02.019153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.005994    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"
Oct 30 21:22:02.034156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.034113    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxlr\" (UniqueName: \"kubernetes.io/projected/536e37a1-058c-42e5-8b80-7cc2d4194711-kube-api-access-jrxlr\") pod \"payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k\" (UID: \"536e37a1-058c-42e5-8b80-7cc2d4194711\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"
Oct 30 21:22:02.034411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.034362    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536e37a1-058c-42e5-8b80-7cc2d4194711-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k\" (UID: \"536e37a1-058c-42e5-8b80-7cc2d4194711\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"
Oct 30 21:22:02.035254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.035229    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536e37a1-058c-42e5-8b80-7cc2d4194711-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k\" (UID: \"536e37a1-058c-42e5-8b80-7cc2d4194711\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"
Oct 30 21:22:02.048902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.048525    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxlr\" (UniqueName: \"kubernetes.io/projected/536e37a1-058c-42e5-8b80-7cc2d4194711-kube-api-access-jrxlr\") pod \"payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k\" (UID: \"536e37a1-058c-42e5-8b80-7cc2d4194711\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"
Oct 30 21:22:02.330520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.315201    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"
Oct 30 21:22:02.366413 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:22:02.359041    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27047a93_6fc0_4006_80a4_dbef6512129b.slice/crio-be3320fa866602218483ebda79ffece540165b3248cbf83ce590791c9d4ea01e WatchSource:0}: Error finding container be3320fa866602218483ebda79ffece540165b3248cbf83ce590791c9d4ea01e: Status 404 returned error can't find the container with id be3320fa866602218483ebda79ffece540165b3248cbf83ce590791c9d4ea01e
Oct 30 21:22:02.371062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.370793    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"]
Oct 30 21:22:02.399664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.399302    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"]
Oct 30 21:22:02.548130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.548092    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w" event={"ID":"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae","Type":"ContainerStarted","Data":"d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b"}
Oct 30 21:22:02.548294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.548158    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w" event={"ID":"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae","Type":"ContainerStarted","Data":"d9dcdfe5964e5cc04ea05d9399f8ed167eb12406229e7eb760997763b334d696"}
Oct 30 21:22:02.548294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.548194    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"
Oct 30 21:22:02.549990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.549961    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn" event={"ID":"27047a93-6fc0-4006-80a4-dbef6512129b","Type":"ContainerStarted","Data":"7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394"}
Oct 30 21:22:02.550104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.550006    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn" event={"ID":"27047a93-6fc0-4006-80a4-dbef6512129b","Type":"ContainerStarted","Data":"be3320fa866602218483ebda79ffece540165b3248cbf83ce590791c9d4ea01e"}
Oct 30 21:22:02.550104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.550028    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"
Oct 30 21:22:02.572040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.571997    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w" podStartSLOduration=1.5719845829999999 podStartE2EDuration="1.571984583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:02.568149309 +0000 UTC m=+1235.614708470" watchObservedRunningTime="2026-10-30 21:22:02.571984583 +0000 UTC m=+1235.618543727"
Oct 30 21:22:02.587123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.587070    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn" podStartSLOduration=1.587058758 podStartE2EDuration="1.587058758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:02.586557834 +0000 UTC m=+1235.633116979" watchObservedRunningTime="2026-10-30 21:22:02.587058758 +0000 UTC m=+1235.633617904"
Oct 30 21:22:02.631689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.631659    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"]
Oct 30 21:22:02.636538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.636516    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"]
Oct 30 21:22:02.637754 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:22:02.637736    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536e37a1_058c_42e5_8b80_7cc2d4194711.slice/crio-ef7fd49c9bac1390ea9dbea722ad317cd226297c67de6a1bef0b426547c57260 WatchSource:0}: Error finding container ef7fd49c9bac1390ea9dbea722ad317cd226297c67de6a1bef0b426547c57260: Status 404 returned error can't find the container with id ef7fd49c9bac1390ea9dbea722ad317cd226297c67de6a1bef0b426547c57260
Oct 30 21:22:02.655018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:02.654986    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"]
Oct 30 21:22:03.559754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:03.559715    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k" event={"ID":"536e37a1-058c-42e5-8b80-7cc2d4194711","Type":"ContainerStarted","Data":"96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa"}
Oct 30 21:22:03.560132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:03.559787    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"
Oct 30 21:22:03.560132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:03.559808    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k" event={"ID":"536e37a1-058c-42e5-8b80-7cc2d4194711","Type":"ContainerStarted","Data":"ef7fd49c9bac1390ea9dbea722ad317cd226297c67de6a1bef0b426547c57260"}
Oct 30 21:22:03.581072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:03.581023    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k" podStartSLOduration=2.581009201 podStartE2EDuration="2.581009201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:03.57784379 +0000 UTC m=+1236.624402956" watchObservedRunningTime="2026-10-30 21:22:03.581009201 +0000 UTC m=+1236.627568346"
Oct 30 21:22:07.632130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:07.632076    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7" podUID="9c9a6fd3-8f46-42da-9ca1-4906ce85b1af" containerName="payload-processing" containerID="cri-o://3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe" gracePeriod=25
Oct 30 21:22:07.655900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:07.655852    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv" podUID="fb1829fc-690b-4568-9a52-3e0b5344cd26" containerName="payload-pre-processing" containerID="cri-o://8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343" gracePeriod=25
Oct 30 21:22:07.964367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:07.964339    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"
Oct 30 21:22:07.997591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:07.997561    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"
Oct 30 21:22:08.051723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.051686    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-plugins-config-volume\") pod \"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af\" (UID: \"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af\") "
Oct 30 21:22:08.051955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.051755    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-kube-api-access-fjhj5\" (UniqueName: \"kubernetes.io/projected/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-kube-api-access-fjhj5\") pod \"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af\" (UID: \"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af\") "
Oct 30 21:22:08.051955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.051867    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1829fc-690b-4568-9a52-3e0b5344cd26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1829fc-690b-4568-9a52-3e0b5344cd26-plugins-config-volume\") pod \"fb1829fc-690b-4568-9a52-3e0b5344cd26\" (UID: \"fb1829fc-690b-4568-9a52-3e0b5344cd26\") "
Oct 30 21:22:08.051955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.051939    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb1829fc-690b-4568-9a52-3e0b5344cd26-kube-api-access-jqb9g\" (UniqueName: \"kubernetes.io/projected/fb1829fc-690b-4568-9a52-3e0b5344cd26-kube-api-access-jqb9g\") pod \"fb1829fc-690b-4568-9a52-3e0b5344cd26\" (UID: \"fb1829fc-690b-4568-9a52-3e0b5344cd26\") "
Oct 30 21:22:08.052594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.052207    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-plugins-config-volume" pod "9c9a6fd3-8f46-42da-9ca1-4906ce85b1af" (UID: "9c9a6fd3-8f46-42da-9ca1-4906ce85b1af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:08.052594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.052283    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1829fc-690b-4568-9a52-3e0b5344cd26-plugins-config-volume" pod "fb1829fc-690b-4568-9a52-3e0b5344cd26" (UID: "fb1829fc-690b-4568-9a52-3e0b5344cd26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:08.052774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.052627    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:08.052774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.052650    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1829fc-690b-4568-9a52-3e0b5344cd26-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:08.054562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.054539    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1829fc-690b-4568-9a52-3e0b5344cd26-kube-api-access-jqb9g" pod "fb1829fc-690b-4568-9a52-3e0b5344cd26" (UID: "fb1829fc-690b-4568-9a52-3e0b5344cd26"). InnerVolumeSpecName "kube-api-access-jqb9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:08.054800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.054783    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-kube-api-access-fjhj5" pod "9c9a6fd3-8f46-42da-9ca1-4906ce85b1af" (UID: "9c9a6fd3-8f46-42da-9ca1-4906ce85b1af"). InnerVolumeSpecName "kube-api-access-fjhj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:08.154324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.154290    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjhj5\" (UniqueName: \"kubernetes.io/projected/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af-kube-api-access-fjhj5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:08.154324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.154327    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqb9g\" (UniqueName: \"kubernetes.io/projected/fb1829fc-690b-4568-9a52-3e0b5344cd26-kube-api-access-jqb9g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:08.364487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.364460    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"
Oct 30 21:22:08.364684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.364541    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"
Oct 30 21:22:08.613574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.613535    4918 generic.go:356] "Generic (PLEG): container finished" podID="fb1829fc-690b-4568-9a52-3e0b5344cd26" containerID="8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343" exitCode=0
Oct 30 21:22:08.613772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.613584    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"
Oct 30 21:22:08.613772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.613619    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv" event={"ID":"fb1829fc-690b-4568-9a52-3e0b5344cd26","Type":"ContainerDied","Data":"8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343"}
Oct 30 21:22:08.613772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.613679    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv" event={"ID":"fb1829fc-690b-4568-9a52-3e0b5344cd26","Type":"ContainerDied","Data":"0b4b4423f9acec4ca52510804aac554ccf79b7c73df3dc870d9ee421bb32005e"}
Oct 30 21:22:08.613772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.613708    4918 scope.go:122] "RemoveContainer" containerID="8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343"
Oct 30 21:22:08.615517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.615493    4918 generic.go:356] "Generic (PLEG): container finished" podID="9c9a6fd3-8f46-42da-9ca1-4906ce85b1af" containerID="3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe" exitCode=0
Oct 30 21:22:08.615646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.615545    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"
Oct 30 21:22:08.615757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.615730    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7" event={"ID":"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af","Type":"ContainerDied","Data":"3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe"}
Oct 30 21:22:08.615848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.615787    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7" event={"ID":"9c9a6fd3-8f46-42da-9ca1-4906ce85b1af","Type":"ContainerDied","Data":"aa8bbc5567ecce293182db122332fd1f659c341363ec752e653ca80a217a63b3"}
Oct 30 21:22:08.625935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.625919    4918 scope.go:122] "RemoveContainer" containerID="8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343"
Oct 30 21:22:08.626229 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:08.626208    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343\": container with ID starting with 8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343 not found: ID does not exist" containerID="8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343"
Oct 30 21:22:08.626297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.626240    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343"} err="failed to get container status \"8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343\": rpc error: code = NotFound desc = could not find container \"8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343\": container with ID starting with 8ba66c51a2f8c71fb9ab3ae047a5e61a36b17b62de01be0bce7aafa7f27f2343 not found: ID does not exist"
Oct 30 21:22:08.626297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.626262    4918 scope.go:122] "RemoveContainer" containerID="3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe"
Oct 30 21:22:08.636332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.636220    4918 scope.go:122] "RemoveContainer" containerID="3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe"
Oct 30 21:22:08.636632 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:08.636526    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe\": container with ID starting with 3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe not found: ID does not exist" containerID="3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe"
Oct 30 21:22:08.636632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.636551    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe"} err="failed to get container status \"3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe\": rpc error: code = NotFound desc = could not find container \"3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe\": container with ID starting with 3aac2ab7456203f473f8b2d014c74566496ba234f7ea4c2a58c2aef89f9fffbe not found: ID does not exist"
Oct 30 21:22:08.642537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.642518    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"]
Oct 30 21:22:08.646371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.646345    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-dbc08f-5f8d4bc7ff-c9nrv"]
Oct 30 21:22:08.657707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.657683    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"]
Oct 30 21:22:08.660918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:08.660899    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dbc08f-6fc746ddfc-fg8n7"]
Oct 30 21:22:09.081137 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:09.081110    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9a6fd3-8f46-42da-9ca1-4906ce85b1af" path="/var/lib/kubelet/pods/9c9a6fd3-8f46-42da-9ca1-4906ce85b1af/volumes"
Oct 30 21:22:09.081627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:09.081602    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb1829fc-690b-4568-9a52-3e0b5344cd26" path="/var/lib/kubelet/pods/fb1829fc-690b-4568-9a52-3e0b5344cd26/volumes"
Oct 30 21:22:09.996698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:09.996668    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"]
Oct 30 21:22:09.997378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:09.997364    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9c9a6fd3-8f46-42da-9ca1-4906ce85b1af" containerName="payload-processing"
Oct 30 21:22:09.997448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:09.997379    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9a6fd3-8f46-42da-9ca1-4906ce85b1af" containerName="payload-processing"
Oct 30 21:22:09.997448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:09.997432    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fb1829fc-690b-4568-9a52-3e0b5344cd26" containerName="payload-pre-processing"
Oct 30 21:22:09.997448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:09.997439    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb1829fc-690b-4568-9a52-3e0b5344cd26" containerName="payload-pre-processing"
Oct 30 21:22:09.997547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:09.997538    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb1829fc-690b-4568-9a52-3e0b5344cd26" containerName="payload-pre-processing"
Oct 30 21:22:09.997579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:09.997558    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9a6fd3-8f46-42da-9ca1-4906ce85b1af" containerName="payload-processing"
Oct 30 21:22:10.002717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.002699    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.014943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.014921    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"]
Oct 30 21:22:10.110654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.110620    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.110654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.110656    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de805cdc-648b-4f04-89e5-d39079fffd6b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.110854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.110791    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.111020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.110999    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.111141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.111110    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzvwr\" (UniqueName: \"kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-kube-api-access-hzvwr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.111287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.111254    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.111367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.111322    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.212780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.212742    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.212962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.212852    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzvwr\" (UniqueName: \"kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-kube-api-access-hzvwr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.212962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.212898    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.212962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.212918    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.212962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.212959    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.213200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.212985    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de805cdc-648b-4f04-89e5-d39079fffd6b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.213200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.213103    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.213975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.213947    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.215454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.215425    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.215551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.215511    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.215618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.215553    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.215725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.215709    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de805cdc-648b-4f04-89e5-d39079fffd6b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.226483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.226456    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.227604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.227586    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzvwr\" (UniqueName: \"kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-kube-api-access-hzvwr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hvrwv\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.319217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.319137    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:10.723943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.723913    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"]
Oct 30 21:22:10.724084 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:22:10.723962    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde805cdc_648b_4f04_89e5_d39079fffd6b.slice/crio-2dd1531b2c41910f5705b251b68118b4fba87f92011060d5b5c7bdefcef321da WatchSource:0}: Error finding container 2dd1531b2c41910f5705b251b68118b4fba87f92011060d5b5c7bdefcef321da: Status 404 returned error can't find the container with id 2dd1531b2c41910f5705b251b68118b4fba87f92011060d5b5c7bdefcef321da
Oct 30 21:22:10.726689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.726658    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:22:10.726774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:10.726718    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:22:11.713414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:11.708555    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv" event={"ID":"de805cdc-648b-4f04-89e5-d39079fffd6b","Type":"ContainerStarted","Data":"07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864"}
Oct 30 21:22:11.713414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:11.708633    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv" event={"ID":"de805cdc-648b-4f04-89e5-d39079fffd6b","Type":"ContainerStarted","Data":"2dd1531b2c41910f5705b251b68118b4fba87f92011060d5b5c7bdefcef321da"}
Oct 30 21:22:11.713414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:11.710155    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:11.723911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:11.723592    4918 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hvrwv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:22:11.723911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:11.723644    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv" podUID="de805cdc-648b-4f04-89e5-d39079fffd6b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:22:11.747959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:11.747907    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv" podStartSLOduration=2.747892812 podStartE2EDuration="2.747892812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:11.743571147 +0000 UTC m=+1244.790130293" watchObservedRunningTime="2026-10-30 21:22:11.747892812 +0000 UTC m=+1244.794451958"
Oct 30 21:22:12.717911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:12.717885    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:22:13.575556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:13.575527    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"
Oct 30 21:22:13.587775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:13.587746    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"
Oct 30 21:22:14.570272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:14.570245    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"
Oct 30 21:22:16.478065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.478030    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"]
Oct 30 21:22:16.482327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.482311    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:16.490573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.490340    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:22:16.490573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.490442    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:22:16.490573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.490464    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:22:16.490573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.490547    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:22:16.495064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.495039    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"]
Oct 30 21:22:16.573578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.573480    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftts6\" (UniqueName: \"kubernetes.io/projected/7ebf4134-3276-4429-803b-9a63ec698c42-kube-api-access-ftts6\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg\" (UID: \"7ebf4134-3276-4429-803b-9a63ec698c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:16.575778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.575751    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg\" (UID: \"7ebf4134-3276-4429-803b-9a63ec698c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:16.582424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.576026    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg\" (UID: \"7ebf4134-3276-4429-803b-9a63ec698c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:16.684552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.684509    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftts6\" (UniqueName: \"kubernetes.io/projected/7ebf4134-3276-4429-803b-9a63ec698c42-kube-api-access-ftts6\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg\" (UID: \"7ebf4134-3276-4429-803b-9a63ec698c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:16.685371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.685346    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg\" (UID: \"7ebf4134-3276-4429-803b-9a63ec698c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:16.692423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.686500    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg\" (UID: \"7ebf4134-3276-4429-803b-9a63ec698c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:16.693243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.686322    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg\" (UID: \"7ebf4134-3276-4429-803b-9a63ec698c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:16.703906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.703852    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg\" (UID: \"7ebf4134-3276-4429-803b-9a63ec698c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:16.716151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.716121    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftts6\" (UniqueName: \"kubernetes.io/projected/7ebf4134-3276-4429-803b-9a63ec698c42-kube-api-access-ftts6\") pod \"maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg\" (UID: \"7ebf4134-3276-4429-803b-9a63ec698c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:16.799571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:16.799485    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:17.098705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:17.098678    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"]
Oct 30 21:22:17.099082 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:22:17.099041    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebf4134_3276_4429_803b_9a63ec698c42.slice/crio-1343610ee5f82c43fc9bc3ed6817c28110b3a653c21d18b1e8d5efbf966a6790 WatchSource:0}: Error finding container 1343610ee5f82c43fc9bc3ed6817c28110b3a653c21d18b1e8d5efbf966a6790: Status 404 returned error can't find the container with id 1343610ee5f82c43fc9bc3ed6817c28110b3a653c21d18b1e8d5efbf966a6790
Oct 30 21:22:17.790016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:17.789981    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg" event={"ID":"7ebf4134-3276-4429-803b-9a63ec698c42","Type":"ContainerStarted","Data":"baa974c152096eea6cbdc29c958a198f8cdfff7b6ecbc0d54b7c62522da4faa1"}
Oct 30 21:22:17.790563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:17.790036    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg" event={"ID":"7ebf4134-3276-4429-803b-9a63ec698c42","Type":"ContainerStarted","Data":"1343610ee5f82c43fc9bc3ed6817c28110b3a653c21d18b1e8d5efbf966a6790"}
Oct 30 21:22:17.816511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:17.816464    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg" podStartSLOduration=1.8164502900000001 podStartE2EDuration="1.81645029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:17.811679259 +0000 UTC m=+1250.858238421" watchObservedRunningTime="2026-10-30 21:22:17.81645029 +0000 UTC m=+1250.863009435"
Oct 30 21:22:18.820870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:18.820836    4918 generic.go:356] "Generic (PLEG): container finished" podID="7ebf4134-3276-4429-803b-9a63ec698c42" containerID="baa974c152096eea6cbdc29c958a198f8cdfff7b6ecbc0d54b7c62522da4faa1" exitCode=0
Oct 30 21:22:18.821998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:18.821805    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg" event={"ID":"7ebf4134-3276-4429-803b-9a63ec698c42","Type":"ContainerDied","Data":"baa974c152096eea6cbdc29c958a198f8cdfff7b6ecbc0d54b7c62522da4faa1"}
Oct 30 21:22:20.008696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.008668    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:20.128775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.128692    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-xks-ca\") pod \"7ebf4134-3276-4429-803b-9a63ec698c42\" (UID: \"7ebf4134-3276-4429-803b-9a63ec698c42\") "
Oct 30 21:22:20.128934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.128841    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-openshift-ca\") pod \"7ebf4134-3276-4429-803b-9a63ec698c42\" (UID: \"7ebf4134-3276-4429-803b-9a63ec698c42\") "
Oct 30 21:22:20.129005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.128951    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebf4134-3276-4429-803b-9a63ec698c42-kube-api-access-ftts6\" (UniqueName: \"kubernetes.io/projected/7ebf4134-3276-4429-803b-9a63ec698c42-kube-api-access-ftts6\") pod \"7ebf4134-3276-4429-803b-9a63ec698c42\" (UID: \"7ebf4134-3276-4429-803b-9a63ec698c42\") "
Oct 30 21:22:20.129252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.129216    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-openshift-ca" pod "7ebf4134-3276-4429-803b-9a63ec698c42" (UID: "7ebf4134-3276-4429-803b-9a63ec698c42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:20.130287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.130258    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:20.131290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.131263    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-xks-ca" pod "7ebf4134-3276-4429-803b-9a63ec698c42" (UID: "7ebf4134-3276-4429-803b-9a63ec698c42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:20.131447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.131370    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebf4134-3276-4429-803b-9a63ec698c42-kube-api-access-ftts6" pod "7ebf4134-3276-4429-803b-9a63ec698c42" (UID: "7ebf4134-3276-4429-803b-9a63ec698c42"). InnerVolumeSpecName "kube-api-access-ftts6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:20.244419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.239076    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ebf4134-3276-4429-803b-9a63ec698c42-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:20.244419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.239121    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftts6\" (UniqueName: \"kubernetes.io/projected/7ebf4134-3276-4429-803b-9a63ec698c42-kube-api-access-ftts6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:20.646734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.646694    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"]
Oct 30 21:22:20.647251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.647218    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" podUID="5f719e78-068e-4c52-96c5-58eddc3de968" containerName="istio-proxy" containerID="cri-o://6969b30b94575aaf43fa8c16fed170bbef6b68dccc1546a08d7fb450c3ec6bc1" gracePeriod=30
Oct 30 21:22:20.757332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.757275    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"]
Oct 30 21:22:20.760761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.760725    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" podUID="ae65e4e9-386b-440c-b568-2402f61346b4" containerName="istio-proxy" containerID="cri-o://a5c5c7f68ff7db45cf1f20fe8a2fa57dcc30f9d3df4b5d07b0121ac0ae418883" gracePeriod=30
Oct 30 21:22:20.851693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.851661    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"
Oct 30 21:22:20.851906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.851875    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg" event={"ID":"7ebf4134-3276-4429-803b-9a63ec698c42","Type":"ContainerDied","Data":"1343610ee5f82c43fc9bc3ed6817c28110b3a653c21d18b1e8d5efbf966a6790"}
Oct 30 21:22:20.851966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:20.851935    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1343610ee5f82c43fc9bc3ed6817c28110b3a653c21d18b1e8d5efbf966a6790"
Oct 30 21:22:24.514693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:24.514668    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9_6012c0fa-471a-41ec-ba1a-95be1768af03/payload-processing/0.log"
Oct 30 21:22:25.247481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:25.247451    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9_9a52b55e-ed84-4a7e-9a72-d001e7e116e7/payload-processing/0.log"
Oct 30 21:22:25.466196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:25.466170    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9_9a52b55e-ed84-4a7e-9a72-d001e7e116e7/payload-processing/0.log"
Oct 30 21:22:25.942514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:25.942478    4918 generic.go:356] "Generic (PLEG): container finished" podID="5f719e78-068e-4c52-96c5-58eddc3de968" containerID="6969b30b94575aaf43fa8c16fed170bbef6b68dccc1546a08d7fb450c3ec6bc1" exitCode=0
Oct 30 21:22:25.942925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:25.942790    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" event={"ID":"5f719e78-068e-4c52-96c5-58eddc3de968","Type":"ContainerDied","Data":"6969b30b94575aaf43fa8c16fed170bbef6b68dccc1546a08d7fb450c3ec6bc1"}
Oct 30 21:22:25.944839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:25.944810    4918 generic.go:356] "Generic (PLEG): container finished" podID="ae65e4e9-386b-440c-b568-2402f61346b4" containerID="a5c5c7f68ff7db45cf1f20fe8a2fa57dcc30f9d3df4b5d07b0121ac0ae418883" exitCode=0
Oct 30 21:22:25.944964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:25.944938    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" event={"ID":"ae65e4e9-386b-440c-b568-2402f61346b4","Type":"ContainerDied","Data":"a5c5c7f68ff7db45cf1f20fe8a2fa57dcc30f9d3df4b5d07b0121ac0ae418883"}
Oct 30 21:22:26.114292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.114266    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:22:26.116289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.116271    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:22:26.233605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.232214    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-kube-api-access-t4x4c\" (UniqueName: \"kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-kube-api-access-t4x4c\") pod \"ae65e4e9-386b-440c-b568-2402f61346b4\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") "
Oct 30 21:22:26.233605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.232277    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-socket\") pod \"ae65e4e9-386b-440c-b568-2402f61346b4\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") "
Oct 30 21:22:26.233605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.232617    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-certs\") pod \"ae65e4e9-386b-440c-b568-2402f61346b4\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") "
Oct 30 21:22:26.233605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.232656    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-socket\") pod \"5f719e78-068e-4c52-96c5-58eddc3de968\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") "
Oct 30 21:22:26.233605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.233573    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-kube-api-access-tbml7\" (UniqueName: \"kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-kube-api-access-tbml7\") pod \"5f719e78-068e-4c52-96c5-58eddc3de968\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") "
Oct 30 21:22:26.234031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.233653    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f719e78-068e-4c52-96c5-58eddc3de968-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f719e78-068e-4c52-96c5-58eddc3de968-istiod-ca-cert\") pod \"5f719e78-068e-4c52-96c5-58eddc3de968\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") "
Oct 30 21:22:26.234031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.233735    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-envoy\") pod \"ae65e4e9-386b-440c-b568-2402f61346b4\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") "
Oct 30 21:22:26.234031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.233779    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-data\") pod \"5f719e78-068e-4c52-96c5-58eddc3de968\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") "
Oct 30 21:22:26.234031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.233822    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-credential-socket\") pod \"ae65e4e9-386b-440c-b568-2402f61346b4\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") "
Oct 30 21:22:26.234031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.233902    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae65e4e9-386b-440c-b568-2402f61346b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae65e4e9-386b-440c-b568-2402f61346b4-istio-podinfo\") pod \"ae65e4e9-386b-440c-b568-2402f61346b4\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") "
Oct 30 21:22:26.234031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.233968    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-data\") pod \"ae65e4e9-386b-440c-b568-2402f61346b4\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") "
Oct 30 21:22:26.234031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.234012    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae65e4e9-386b-440c-b568-2402f61346b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae65e4e9-386b-440c-b568-2402f61346b4-istiod-ca-cert\") pod \"ae65e4e9-386b-440c-b568-2402f61346b4\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") "
Oct 30 21:22:26.234440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.234046    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f719e78-068e-4c52-96c5-58eddc3de968-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f719e78-068e-4c52-96c5-58eddc3de968-istio-podinfo\") pod \"5f719e78-068e-4c52-96c5-58eddc3de968\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") "
Oct 30 21:22:26.234440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.234116    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-istio-token\" (UniqueName: \"kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-istio-token\") pod \"ae65e4e9-386b-440c-b568-2402f61346b4\" (UID: \"ae65e4e9-386b-440c-b568-2402f61346b4\") "
Oct 30 21:22:26.234440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.234160    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-credential-socket\") pod \"5f719e78-068e-4c52-96c5-58eddc3de968\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") "
Oct 30 21:22:26.234440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.234248    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-istio-token\" (UniqueName: \"kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-istio-token\") pod \"5f719e78-068e-4c52-96c5-58eddc3de968\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") "
Oct 30 21:22:26.234440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.234280    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-certs\") pod \"5f719e78-068e-4c52-96c5-58eddc3de968\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") "
Oct 30 21:22:26.234712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.234548    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-envoy\") pod \"5f719e78-068e-4c52-96c5-58eddc3de968\" (UID: \"5f719e78-068e-4c52-96c5-58eddc3de968\") "
Oct 30 21:22:26.235174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.235150    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-credential-socket" pod "ae65e4e9-386b-440c-b568-2402f61346b4" (UID: "ae65e4e9-386b-440c-b568-2402f61346b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:26.236416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.235959    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.236833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.236811    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-data" pod "ae65e4e9-386b-440c-b568-2402f61346b4" (UID: "ae65e4e9-386b-440c-b568-2402f61346b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:26.237337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.237316    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-data" pod "5f719e78-068e-4c52-96c5-58eddc3de968" (UID: "5f719e78-068e-4c52-96c5-58eddc3de968"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:26.237766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.237744    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f719e78-068e-4c52-96c5-58eddc3de968-istiod-ca-cert" pod "5f719e78-068e-4c52-96c5-58eddc3de968" (UID: "5f719e78-068e-4c52-96c5-58eddc3de968"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:26.238990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.238943    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae65e4e9-386b-440c-b568-2402f61346b4-istiod-ca-cert" pod "ae65e4e9-386b-440c-b568-2402f61346b4" (UID: "ae65e4e9-386b-440c-b568-2402f61346b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:26.239615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.239161    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-socket" pod "ae65e4e9-386b-440c-b568-2402f61346b4" (UID: "ae65e4e9-386b-440c-b568-2402f61346b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:26.239615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.239250    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-certs" pod "5f719e78-068e-4c52-96c5-58eddc3de968" (UID: "5f719e78-068e-4c52-96c5-58eddc3de968"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:26.239615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.239485    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-credential-socket" pod "5f719e78-068e-4c52-96c5-58eddc3de968" (UID: "5f719e78-068e-4c52-96c5-58eddc3de968"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:26.239901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.239691    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-socket" pod "5f719e78-068e-4c52-96c5-58eddc3de968" (UID: "5f719e78-068e-4c52-96c5-58eddc3de968"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:26.240170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.240148    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-certs" pod "ae65e4e9-386b-440c-b568-2402f61346b4" (UID: "ae65e4e9-386b-440c-b568-2402f61346b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:26.253805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.253754    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-envoy" pod "5f719e78-068e-4c52-96c5-58eddc3de968" (UID: "5f719e78-068e-4c52-96c5-58eddc3de968"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:26.253805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.253754    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-envoy" pod "ae65e4e9-386b-440c-b568-2402f61346b4" (UID: "ae65e4e9-386b-440c-b568-2402f61346b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:26.253940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.253820    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae65e4e9-386b-440c-b568-2402f61346b4-istio-podinfo" pod "ae65e4e9-386b-440c-b568-2402f61346b4" (UID: "ae65e4e9-386b-440c-b568-2402f61346b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:26.253940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.253828    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-istio-token" pod "ae65e4e9-386b-440c-b568-2402f61346b4" (UID: "ae65e4e9-386b-440c-b568-2402f61346b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:26.253940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.253861    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-kube-api-access-tbml7" pod "5f719e78-068e-4c52-96c5-58eddc3de968" (UID: "5f719e78-068e-4c52-96c5-58eddc3de968"). InnerVolumeSpecName "kube-api-access-tbml7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:26.255037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.254994    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f719e78-068e-4c52-96c5-58eddc3de968-istio-podinfo" pod "5f719e78-068e-4c52-96c5-58eddc3de968" (UID: "5f719e78-068e-4c52-96c5-58eddc3de968"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:26.255259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.255235    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-kube-api-access-t4x4c" pod "ae65e4e9-386b-440c-b568-2402f61346b4" (UID: "ae65e4e9-386b-440c-b568-2402f61346b4"). InnerVolumeSpecName "kube-api-access-t4x4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:26.257282 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.257252    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-istio-token" pod "5f719e78-068e-4c52-96c5-58eddc3de968" (UID: "5f719e78-068e-4c52-96c5-58eddc3de968"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:26.337172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337104    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae65e4e9-386b-440c-b568-2402f61346b4-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337132    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337141    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae65e4e9-386b-440c-b568-2402f61346b4-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337151    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f719e78-068e-4c52-96c5-58eddc3de968-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337159    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337168    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337176    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337184    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337192    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337200    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4x4c\" (UniqueName: \"kubernetes.io/projected/ae65e4e9-386b-440c-b568-2402f61346b4-kube-api-access-t4x4c\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337209    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337219    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337228    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337235    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbml7\" (UniqueName: \"kubernetes.io/projected/5f719e78-068e-4c52-96c5-58eddc3de968-kube-api-access-tbml7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337243    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f719e78-068e-4c52-96c5-58eddc3de968-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337251    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae65e4e9-386b-440c-b568-2402f61346b4-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.337564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.337259    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f719e78-068e-4c52-96c5-58eddc3de968-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:26.669763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.669689    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"]
Oct 30 21:22:26.670878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.670678    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5f719e78-068e-4c52-96c5-58eddc3de968" containerName="istio-proxy"
Oct 30 21:22:26.671147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.671129    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f719e78-068e-4c52-96c5-58eddc3de968" containerName="istio-proxy"
Oct 30 21:22:26.671330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.671318    4918 cpu_manager.go:414] "containerMap: removing container" podUID="7ebf4134-3276-4429-803b-9a63ec698c42" containerName="revoke-keys"
Oct 30 21:22:26.671460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.671448    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ebf4134-3276-4429-803b-9a63ec698c42" containerName="revoke-keys"
Oct 30 21:22:26.671574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.671547    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ae65e4e9-386b-440c-b568-2402f61346b4" containerName="istio-proxy"
Oct 30 21:22:26.671659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.671650    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae65e4e9-386b-440c-b568-2402f61346b4" containerName="istio-proxy"
Oct 30 21:22:26.671908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.671898    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f719e78-068e-4c52-96c5-58eddc3de968" containerName="istio-proxy"
Oct 30 21:22:26.672034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.672025    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ebf4134-3276-4429-803b-9a63ec698c42" containerName="revoke-keys"
Oct 30 21:22:26.672103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.672084    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae65e4e9-386b-440c-b568-2402f61346b4" containerName="istio-proxy"
Oct 30 21:22:26.683198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.680951    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:26.684235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.684201    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"]
Oct 30 21:22:26.691055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.690247    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:22:26.691055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.690313    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:22:26.691055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.690452    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:22:26.691055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.690512    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:22:26.731100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.731067    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq\" (UID: \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:26.731258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.731185    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5z8g\" (UniqueName: \"kubernetes.io/projected/45addcc7-b265-418c-bbbd-f9a2a275d8c5-kube-api-access-s5z8g\") pod \"maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq\" (UID: \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:26.731300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.731289    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq\" (UID: \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:26.833026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.832487    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq\" (UID: \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:26.833026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.832681    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq\" (UID: \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:26.833026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.832774    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5z8g\" (UniqueName: \"kubernetes.io/projected/45addcc7-b265-418c-bbbd-f9a2a275d8c5-kube-api-access-s5z8g\") pod \"maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq\" (UID: \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:26.834556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.834525    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"]
Oct 30 21:22:26.834769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.834741    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq\" (UID: \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:26.840133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.840113    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:26.840782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.840757    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq\" (UID: \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:26.849973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.848375    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-80e936-openshift-default-dockercfg-jnhmt\""
Oct 30 21:22:26.862857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.862833    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"]
Oct 30 21:22:26.863442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:26.863381    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5z8g\" (UniqueName: \"kubernetes.io/projected/45addcc7-b265-418c-bbbd-f9a2a275d8c5-kube-api-access-s5z8g\") pod \"maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq\" (UID: \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:27.041447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.021731    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:27.041447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.038606    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt" event={"ID":"ae65e4e9-386b-440c-b568-2402f61346b4","Type":"ContainerDied","Data":"2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f"}
Oct 30 21:22:27.041447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.038713    4918 scope.go:122] "RemoveContainer" containerID="a5c5c7f68ff7db45cf1f20fe8a2fa57dcc30f9d3df4b5d07b0121ac0ae418883"
Oct 30 21:22:27.043666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.043639    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"
Oct 30 21:22:27.101416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.088269    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-envoy\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.101416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.088334    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-socket\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.101416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.088378    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020138d9-94c2-4a2f-98a6-f92f25f661ee-istiod-ca-cert\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.101416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.088490    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-token\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.101416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.088527    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-podinfo\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.101416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.088594    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-data\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.101416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.088652    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-credential-socket\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.101416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.088702    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9zmj\" (UniqueName: \"kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-kube-api-access-l9zmj\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.101416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.088823    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-certs\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.198416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.197779    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-envoy\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.198416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.197847    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-socket\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.198416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.197883    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020138d9-94c2-4a2f-98a6-f92f25f661ee-istiod-ca-cert\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.198416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.197963    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-token\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.198416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.197986    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-podinfo\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.198416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.198026    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-data\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.198416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.198067    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-credential-socket\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.198416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.198111    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zmj\" (UniqueName: \"kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-kube-api-access-l9zmj\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.198416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.198228    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-certs\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.199023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.198665    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-certs\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.223196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.204422    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-data\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.223196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.205635    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-credential-socket\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.223196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.206035    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-socket\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.223196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.207265    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020138d9-94c2-4a2f-98a6-f92f25f661ee-istiod-ca-cert\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.223196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.212459    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"]
Oct 30 21:22:27.223196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.212493    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-7wpjt"]
Oct 30 21:22:27.247312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.247284    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-envoy\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.250803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.248288    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-token\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.250803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.248734    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9zmj\" (UniqueName: \"kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-kube-api-access-l9zmj\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.250803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.248830    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc" event={"ID":"5f719e78-068e-4c52-96c5-58eddc3de968","Type":"ContainerDied","Data":"e9879d248f6094918389effbee441d6371327ce09625f43417add182ac877cf7"}
Oct 30 21:22:27.250803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.248985    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"
Oct 30 21:22:27.256078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.256045    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-podinfo\") pod \"e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") " pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.292316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.292240    4918 scope.go:122] "RemoveContainer" containerID="6969b30b94575aaf43fa8c16fed170bbef6b68dccc1546a08d7fb450c3ec6bc1"
Oct 30 21:22:27.306807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.306775    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"]
Oct 30 21:22:27.311005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.310981    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-t4vwc"]
Oct 30 21:22:27.512967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.512937    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-80e936-openshift-default-dockercfg-jnhmt\""
Oct 30 21:22:27.514882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.514856    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:27.565871 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:22:27.565835    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45addcc7_b265_418c_bbbd_f9a2a275d8c5.slice/crio-13610ddb674a2c948c39b6552928f578e638ce479b558c5f00bdbddea73ea1af WatchSource:0}: Error finding container 13610ddb674a2c948c39b6552928f578e638ce479b558c5f00bdbddea73ea1af: Status 404 returned error can't find the container with id 13610ddb674a2c948c39b6552928f578e638ce479b558c5f00bdbddea73ea1af
Oct 30 21:22:27.566100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.566078    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"]
Oct 30 21:22:27.870541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.870380    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"]
Oct 30 21:22:27.897705 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:22:27.897665    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020138d9_94c2_4a2f_98a6_f92f25f661ee.slice/crio-c0aedd3f4ab939cab30b895f17fb61e82b0a66a45e2d5a7b75459c2e697c230e WatchSource:0}: Error finding container c0aedd3f4ab939cab30b895f17fb61e82b0a66a45e2d5a7b75459c2e697c230e: Status 404 returned error can't find the container with id c0aedd3f4ab939cab30b895f17fb61e82b0a66a45e2d5a7b75459c2e697c230e
Oct 30 21:22:27.905371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.904103    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:22:27.905766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.905737    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:22:27.905976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:27.905959    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:22:28.263030 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:28.262996    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq" event={"ID":"45addcc7-b265-418c-bbbd-f9a2a275d8c5","Type":"ContainerStarted","Data":"be378ec323ee9c167a0cf61c3f78268ae6397a501d924a1bec2febafb8cb776a"}
Oct 30 21:22:28.263562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:28.263054    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq" event={"ID":"45addcc7-b265-418c-bbbd-f9a2a275d8c5","Type":"ContainerStarted","Data":"13610ddb674a2c948c39b6552928f578e638ce479b558c5f00bdbddea73ea1af"}
Oct 30 21:22:28.265066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:28.265039    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" event={"ID":"020138d9-94c2-4a2f-98a6-f92f25f661ee","Type":"ContainerStarted","Data":"66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c"}
Oct 30 21:22:28.265220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:28.265084    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" event={"ID":"020138d9-94c2-4a2f-98a6-f92f25f661ee","Type":"ContainerStarted","Data":"c0aedd3f4ab939cab30b895f17fb61e82b0a66a45e2d5a7b75459c2e697c230e"}
Oct 30 21:22:28.283372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:28.283327    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq" podStartSLOduration=2.283312828 podStartE2EDuration="2.283312828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:28.280761055 +0000 UTC m=+1261.327320219" watchObservedRunningTime="2026-10-30 21:22:28.283312828 +0000 UTC m=+1261.329871973"
Oct 30 21:22:28.306539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:28.306486    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" podStartSLOduration=2.306474413 podStartE2EDuration="2.306474413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:28.302420752 +0000 UTC m=+1261.348979922" watchObservedRunningTime="2026-10-30 21:22:28.306474413 +0000 UTC m=+1261.353033557"
Oct 30 21:22:28.513248 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:28.513164    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:28.515856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:28.515830    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:28.521755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:28.521705    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:29.078575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.078545    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f719e78-068e-4c52-96c5-58eddc3de968" path="/var/lib/kubelet/pods/5f719e78-068e-4c52-96c5-58eddc3de968/volumes"
Oct 30 21:22:29.079048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.079034    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae65e4e9-386b-440c-b568-2402f61346b4" path="/var/lib/kubelet/pods/ae65e4e9-386b-440c-b568-2402f61346b4/volumes"
Oct 30 21:22:29.163122 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:29.163083    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:22:29.254262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.254231    4918 scope.go:122] "RemoveContainer" containerID="7fd7338bbc89f1a4813d6cc12e12fef08a947397236f6c7c16af1b9b1aae86b9"
Oct 30 21:22:29.265494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.265471    4918 scope.go:122] "RemoveContainer" containerID="0e39e4a22335917c7248b9e056e586daccf240da131887cab0c7b6789e4e619e"
Oct 30 21:22:29.278960 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.278927    4918 scope.go:122] "RemoveContainer" containerID="cef1e6fa52a62d816845a950be48f257e80c252e8e408992cb7de6210c4f1e4f"
Oct 30 21:22:29.281533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.281503    4918 generic.go:356] "Generic (PLEG): container finished" podID="45addcc7-b265-418c-bbbd-f9a2a275d8c5" containerID="be378ec323ee9c167a0cf61c3f78268ae6397a501d924a1bec2febafb8cb776a" exitCode=0
Oct 30 21:22:29.281634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.281583    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq" event={"ID":"45addcc7-b265-418c-bbbd-f9a2a275d8c5","Type":"ContainerDied","Data":"be378ec323ee9c167a0cf61c3f78268ae6397a501d924a1bec2febafb8cb776a"}
Oct 30 21:22:29.281959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.281921    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:29.282832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.282814    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:22:29.289288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.289270    4918 scope.go:122] "RemoveContainer" containerID="a4a06a0f44a5e76ad613d223f104b86eca5741b33da1526da484525db7cad601"
Oct 30 21:22:29.299337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.299315    4918 scope.go:122] "RemoveContainer" containerID="4b5c203196f232b734b262d7debfff1df2727cf9e842d97f5ed6a6ab1bc0d93e"
Oct 30 21:22:29.310668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.310644    4918 scope.go:122] "RemoveContainer" containerID="1b09c3baf6c3d334bf3266e8f055709eec5b3440a9f62f4bed9822fbd6f57f43"
Oct 30 21:22:29.321571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.321542    4918 scope.go:122] "RemoveContainer" containerID="41c9879a7c7d21df4efd29c74b63be1484ff96e60858d431c92f2fe8eaa9c0f3"
Oct 30 21:22:29.335577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.335507    4918 scope.go:122] "RemoveContainer" containerID="0e84ba44279e00c4bf0c5b8e7da3a7daa9850fc08ea53a2326cd7d4a05e0a854"
Oct 30 21:22:29.361609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.361566    4918 scope.go:122] "RemoveContainer" containerID="7c56214b82c2d306af235b9c071b59e820e024bfd3c895c8e74dfe7edfef112b"
Oct 30 21:22:29.373060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.373035    4918 scope.go:122] "RemoveContainer" containerID="f8de87e64b012ae42a2740c5a99d44f7f69453d49545187bd706cb27a6efac9d"
Oct 30 21:22:29.383252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:29.383231    4918 scope.go:122] "RemoveContainer" containerID="4f72bd9a1161f85e8c7a015c314b29d4641c8ccdd4e52467e3c6fc86d8904e51"
Oct 30 21:22:30.645711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:30.645670    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"]
Oct 30 21:22:30.651885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:30.651859    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-dbc08f-dhp6b"]
Oct 30 21:22:30.704984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:30.704961    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:30.804859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:30.804824    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-xks-ca\") pod \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\" (UID: \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\") "
Oct 30 21:22:30.805038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:30.804882    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45addcc7-b265-418c-bbbd-f9a2a275d8c5-kube-api-access-s5z8g\" (UniqueName: \"kubernetes.io/projected/45addcc7-b265-418c-bbbd-f9a2a275d8c5-kube-api-access-s5z8g\") pod \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\" (UID: \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\") "
Oct 30 21:22:30.805038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:30.805026    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-openshift-ca\") pod \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\" (UID: \"45addcc7-b265-418c-bbbd-f9a2a275d8c5\") "
Oct 30 21:22:30.805370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:30.805338    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-openshift-ca" pod "45addcc7-b265-418c-bbbd-f9a2a275d8c5" (UID: "45addcc7-b265-418c-bbbd-f9a2a275d8c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:30.805755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:30.805646    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:30.807133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:30.807111    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-xks-ca" pod "45addcc7-b265-418c-bbbd-f9a2a275d8c5" (UID: "45addcc7-b265-418c-bbbd-f9a2a275d8c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:30.807282 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:30.807269    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45addcc7-b265-418c-bbbd-f9a2a275d8c5-kube-api-access-s5z8g" pod "45addcc7-b265-418c-bbbd-f9a2a275d8c5" (UID: "45addcc7-b265-418c-bbbd-f9a2a275d8c5"). InnerVolumeSpecName "kube-api-access-s5z8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:30.922959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:30.922833    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45addcc7-b265-418c-bbbd-f9a2a275d8c5-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:30.922959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:30.922874    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5z8g\" (UniqueName: \"kubernetes.io/projected/45addcc7-b265-418c-bbbd-f9a2a275d8c5-kube-api-access-s5z8g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:31.092022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:31.091985    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85192bf-e597-4206-8752-6f886fc08ffa" path="/var/lib/kubelet/pods/d85192bf-e597-4206-8752-6f886fc08ffa/volumes"
Oct 30 21:22:31.297581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:31.297557    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"
Oct 30 21:22:31.297752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:31.297587    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq" event={"ID":"45addcc7-b265-418c-bbbd-f9a2a275d8c5","Type":"ContainerDied","Data":"13610ddb674a2c948c39b6552928f578e638ce479b558c5f00bdbddea73ea1af"}
Oct 30 21:22:31.297752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:31.297632    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13610ddb674a2c948c39b6552928f578e638ce479b558c5f00bdbddea73ea1af"
Oct 30 21:22:31.880811 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:31.880771    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:38.832687 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:38.832641    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:39.012416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.011511    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"]
Oct 30 21:22:39.019419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.012670    4918 cpu_manager.go:414] "containerMap: removing container" podUID="45addcc7-b265-418c-bbbd-f9a2a275d8c5" containerName="revoke-keys"
Oct 30 21:22:39.019419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.012699    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45addcc7-b265-418c-bbbd-f9a2a275d8c5" containerName="revoke-keys"
Oct 30 21:22:39.019419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.012921    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="45addcc7-b265-418c-bbbd-f9a2a275d8c5" containerName="revoke-keys"
Oct 30 21:22:39.082415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.077774    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:39.096493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.090482    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:22:39.096493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.090564    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:22:39.096493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.090673    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:22:39.096493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.094101    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"]
Oct 30 21:22:39.122122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.119083    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:22:39.173621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.173591    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"]
Oct 30 21:22:39.174069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.174030    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy" containerID="cri-o://628b293d70552b7591bf94a822b9c14394cc0dba5f8aa6678bd9c9dfd42c1286" gracePeriod=30
Oct 30 21:22:39.180767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.180741    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"]
Oct 30 21:22:39.181209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.181181    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" podUID="6521302f-c44e-40f0-b342-10c81833e27a" containerName="istio-proxy" containerID="cri-o://8c5cc264f3f94f219169be1bb5546b804acfa72aaaa18905f59fc495f308afc2" gracePeriod=30
Oct 30 21:22:39.210414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.209996    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm\" (UID: \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:39.210414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.210163    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm\" (UID: \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:39.210414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.210238    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twsws\" (UniqueName: \"kubernetes.io/projected/7da1b1bd-9254-4036-b851-d814bbfbaaf3-kube-api-access-twsws\") pod \"maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm\" (UID: \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:39.311804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.311762    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm\" (UID: \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:39.311967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.311918    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm\" (UID: \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:39.312025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.311985    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twsws\" (UniqueName: \"kubernetes.io/projected/7da1b1bd-9254-4036-b851-d814bbfbaaf3-kube-api-access-twsws\") pod \"maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm\" (UID: \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:39.312647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.312618    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm\" (UID: \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:39.314516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.314494    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm\" (UID: \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:39.357372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.357256    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twsws\" (UniqueName: \"kubernetes.io/projected/7da1b1bd-9254-4036-b851-d814bbfbaaf3-kube-api-access-twsws\") pod \"maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm\" (UID: \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:39.458808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.458777    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:39.750979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:39.750952    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"]
Oct 30 21:22:39.752861 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:22:39.752834    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da1b1bd_9254_4036_b851_d814bbfbaaf3.slice/crio-62f20da5195871a865faf325912998d08ae6f0af0948a37de11b926190b6ea2f WatchSource:0}: Error finding container 62f20da5195871a865faf325912998d08ae6f0af0948a37de11b926190b6ea2f: Status 404 returned error can't find the container with id 62f20da5195871a865faf325912998d08ae6f0af0948a37de11b926190b6ea2f
Oct 30 21:22:40.477736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:40.477670    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm" event={"ID":"7da1b1bd-9254-4036-b851-d814bbfbaaf3","Type":"ContainerStarted","Data":"a57f143bb285b8ac0923bb1c3d40b171f73bee39df97c3d8a6c72931e30c09b8"}
Oct 30 21:22:40.478433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:40.477744    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm" event={"ID":"7da1b1bd-9254-4036-b851-d814bbfbaaf3","Type":"ContainerStarted","Data":"62f20da5195871a865faf325912998d08ae6f0af0948a37de11b926190b6ea2f"}
Oct 30 21:22:40.512542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:40.512479    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm" podStartSLOduration=2.512461284 podStartE2EDuration="2.512461284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:40.505973112 +0000 UTC m=+1273.552532251" watchObservedRunningTime="2026-10-30 21:22:40.512461284 +0000 UTC m=+1273.559020430"
Oct 30 21:22:40.564218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:40.564181    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"]
Oct 30 21:22:40.564448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:40.564422    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt" podUID="4094667e-929d-45e3-a0ca-4167b3895ca3" containerName="payload-processing" containerID="cri-o://3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57" gracePeriod=30
Oct 30 21:22:40.579125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:40.577964    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"]
Oct 30 21:22:40.592576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:40.592539    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg" podUID="0985b700-2e5c-4adc-84f9-3102272a76ad" containerName="payload-pre-processing" containerID="cri-o://f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353" gracePeriod=30
Oct 30 21:22:41.153312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.153222    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"]
Oct 30 21:22:41.157019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.156991    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2a55fb-zd6gg"]
Oct 30 21:22:41.239605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.238143    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"
Oct 30 21:22:41.368493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.368468    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"
Oct 30 21:22:41.371136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.371118    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4094667e-929d-45e3-a0ca-4167b3895ca3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4094667e-929d-45e3-a0ca-4167b3895ca3-plugins-config-volume\") pod \"4094667e-929d-45e3-a0ca-4167b3895ca3\" (UID: \"4094667e-929d-45e3-a0ca-4167b3895ca3\") "
Oct 30 21:22:41.371296 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.371281    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4094667e-929d-45e3-a0ca-4167b3895ca3-kube-api-access-69flz\" (UniqueName: \"kubernetes.io/projected/4094667e-929d-45e3-a0ca-4167b3895ca3-kube-api-access-69flz\") pod \"4094667e-929d-45e3-a0ca-4167b3895ca3\" (UID: \"4094667e-929d-45e3-a0ca-4167b3895ca3\") "
Oct 30 21:22:41.371516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.371495    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4094667e-929d-45e3-a0ca-4167b3895ca3-plugins-config-volume" pod "4094667e-929d-45e3-a0ca-4167b3895ca3" (UID: "4094667e-929d-45e3-a0ca-4167b3895ca3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:41.371939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.371920    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4094667e-929d-45e3-a0ca-4167b3895ca3-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:41.374037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.374007    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4094667e-929d-45e3-a0ca-4167b3895ca3-kube-api-access-69flz" pod "4094667e-929d-45e3-a0ca-4167b3895ca3" (UID: "4094667e-929d-45e3-a0ca-4167b3895ca3"). InnerVolumeSpecName "kube-api-access-69flz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:41.472885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.472851    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0985b700-2e5c-4adc-84f9-3102272a76ad-kube-api-access-28d2v\" (UniqueName: \"kubernetes.io/projected/0985b700-2e5c-4adc-84f9-3102272a76ad-kube-api-access-28d2v\") pod \"0985b700-2e5c-4adc-84f9-3102272a76ad\" (UID: \"0985b700-2e5c-4adc-84f9-3102272a76ad\") "
Oct 30 21:22:41.473171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.473153    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0985b700-2e5c-4adc-84f9-3102272a76ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0985b700-2e5c-4adc-84f9-3102272a76ad-plugins-config-volume\") pod \"0985b700-2e5c-4adc-84f9-3102272a76ad\" (UID: \"0985b700-2e5c-4adc-84f9-3102272a76ad\") "
Oct 30 21:22:41.473519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.473497    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0985b700-2e5c-4adc-84f9-3102272a76ad-plugins-config-volume" pod "0985b700-2e5c-4adc-84f9-3102272a76ad" (UID: "0985b700-2e5c-4adc-84f9-3102272a76ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:41.473757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.473744    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69flz\" (UniqueName: \"kubernetes.io/projected/4094667e-929d-45e3-a0ca-4167b3895ca3-kube-api-access-69flz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:41.473813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.473760    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0985b700-2e5c-4adc-84f9-3102272a76ad-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:41.475145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.475124    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0985b700-2e5c-4adc-84f9-3102272a76ad-kube-api-access-28d2v" pod "0985b700-2e5c-4adc-84f9-3102272a76ad" (UID: "0985b700-2e5c-4adc-84f9-3102272a76ad"). InnerVolumeSpecName "kube-api-access-28d2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:41.486789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.486765    4918 generic.go:356] "Generic (PLEG): container finished" podID="0985b700-2e5c-4adc-84f9-3102272a76ad" containerID="f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353" exitCode=0
Oct 30 21:22:41.487161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.486822    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"
Oct 30 21:22:41.487161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.486860    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg" event={"ID":"0985b700-2e5c-4adc-84f9-3102272a76ad","Type":"ContainerDied","Data":"f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353"}
Oct 30 21:22:41.487161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.486916    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg" event={"ID":"0985b700-2e5c-4adc-84f9-3102272a76ad","Type":"ContainerDied","Data":"b68e7ed9c3311a268aa7ae9589bd03449f46f6e6b3d8a52da60e327c34e76762"}
Oct 30 21:22:41.487161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.486945    4918 scope.go:122] "RemoveContainer" containerID="f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353"
Oct 30 21:22:41.489053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.489024    4918 generic.go:356] "Generic (PLEG): container finished" podID="7da1b1bd-9254-4036-b851-d814bbfbaaf3" containerID="a57f143bb285b8ac0923bb1c3d40b171f73bee39df97c3d8a6c72931e30c09b8" exitCode=0
Oct 30 21:22:41.489159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.489074    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm" event={"ID":"7da1b1bd-9254-4036-b851-d814bbfbaaf3","Type":"ContainerDied","Data":"a57f143bb285b8ac0923bb1c3d40b171f73bee39df97c3d8a6c72931e30c09b8"}
Oct 30 21:22:41.490665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.490646    4918 generic.go:356] "Generic (PLEG): container finished" podID="4094667e-929d-45e3-a0ca-4167b3895ca3" containerID="3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57" exitCode=0
Oct 30 21:22:41.490779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.490686    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"
Oct 30 21:22:41.490779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.490694    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt" event={"ID":"4094667e-929d-45e3-a0ca-4167b3895ca3","Type":"ContainerDied","Data":"3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57"}
Oct 30 21:22:41.490779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.490733    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt" event={"ID":"4094667e-929d-45e3-a0ca-4167b3895ca3","Type":"ContainerDied","Data":"329f33f5a4082a59b8cc492dbf63bbac6d827a5ee761c65c1e498596dbb3fbc1"}
Oct 30 21:22:41.524816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.499692    4918 scope.go:122] "RemoveContainer" containerID="f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353"
Oct 30 21:22:41.532421 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:41.528338    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353\": container with ID starting with f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353 not found: ID does not exist" containerID="f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353"
Oct 30 21:22:41.532421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.528381    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353"} err="failed to get container status \"f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353\": rpc error: code = NotFound desc = could not find container \"f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353\": container with ID starting with f69cdd3973dfb91daa0278d8b1758607fdae246641ddecd09664cee36bca8353 not found: ID does not exist"
Oct 30 21:22:41.532421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.528429    4918 scope.go:122] "RemoveContainer" containerID="3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57"
Oct 30 21:22:41.575421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.566352    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"]
Oct 30 21:22:41.612447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.612417    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28d2v\" (UniqueName: \"kubernetes.io/projected/0985b700-2e5c-4adc-84f9-3102272a76ad-kube-api-access-28d2v\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:41.621496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.621461    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2a55fb-57f7db47cf-nptzg"]
Oct 30 21:22:41.627072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.627047    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"]
Oct 30 21:22:41.628934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.628914    4918 scope.go:122] "RemoveContainer" containerID="3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57"
Oct 30 21:22:41.629508 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:41.629483    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57\": container with ID starting with 3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57 not found: ID does not exist" containerID="3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57"
Oct 30 21:22:41.629585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.629518    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57"} err="failed to get container status \"3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57\": rpc error: code = NotFound desc = could not find container \"3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57\": container with ID starting with 3a92e454146bd9cd0cae8886fab1dd6754305725f1f4e4fe623fda7f08e8de57 not found: ID does not exist"
Oct 30 21:22:41.631088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:41.631070    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2a55fb-567d47b67b-qlxxt"]
Oct 30 21:22:42.677589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:42.677556    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:42.832484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:42.830146    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-xks-ca\") pod \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\" (UID: \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\") "
Oct 30 21:22:42.832484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:42.830290    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7da1b1bd-9254-4036-b851-d814bbfbaaf3-kube-api-access-twsws\" (UniqueName: \"kubernetes.io/projected/7da1b1bd-9254-4036-b851-d814bbfbaaf3-kube-api-access-twsws\") pod \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\" (UID: \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\") "
Oct 30 21:22:42.832484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:42.830374    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-openshift-ca\") pod \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\" (UID: \"7da1b1bd-9254-4036-b851-d814bbfbaaf3\") "
Oct 30 21:22:42.832484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:42.831936    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-openshift-ca" pod "7da1b1bd-9254-4036-b851-d814bbfbaaf3" (UID: "7da1b1bd-9254-4036-b851-d814bbfbaaf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:42.851441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:42.843714    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7da1b1bd-9254-4036-b851-d814bbfbaaf3-kube-api-access-twsws" pod "7da1b1bd-9254-4036-b851-d814bbfbaaf3" (UID: "7da1b1bd-9254-4036-b851-d814bbfbaaf3"). InnerVolumeSpecName "kube-api-access-twsws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:42.879415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:42.876620    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-xks-ca" pod "7da1b1bd-9254-4036-b851-d814bbfbaaf3" (UID: "7da1b1bd-9254-4036-b851-d814bbfbaaf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:42.932697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:42.932665    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:42.932697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:42.932694    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twsws\" (UniqueName: \"kubernetes.io/projected/7da1b1bd-9254-4036-b851-d814bbfbaaf3-kube-api-access-twsws\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:42.932697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:42.932704    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7da1b1bd-9254-4036-b851-d814bbfbaaf3-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:43.079441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:43.079375    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0985b700-2e5c-4adc-84f9-3102272a76ad" path="/var/lib/kubelet/pods/0985b700-2e5c-4adc-84f9-3102272a76ad/volumes"
Oct 30 21:22:43.079889 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:43.079874    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4094667e-929d-45e3-a0ca-4167b3895ca3" path="/var/lib/kubelet/pods/4094667e-929d-45e3-a0ca-4167b3895ca3/volumes"
Oct 30 21:22:43.080440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:43.080420    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ebf4134-3276-4429-803b-9a63ec698c42" path="/var/lib/kubelet/pods/7ebf4134-3276-4429-803b-9a63ec698c42/volumes"
Oct 30 21:22:43.518107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:43.518079    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"
Oct 30 21:22:43.518277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:43.518121    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm" event={"ID":"7da1b1bd-9254-4036-b851-d814bbfbaaf3","Type":"ContainerDied","Data":"62f20da5195871a865faf325912998d08ae6f0af0948a37de11b926190b6ea2f"}
Oct 30 21:22:43.518277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:43.518170    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62f20da5195871a865faf325912998d08ae6f0af0948a37de11b926190b6ea2f"
Oct 30 21:22:43.626926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:43.626899    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"]
Oct 30 21:22:43.627104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:43.627085    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k" podUID="0a415d16-9fc3-4b38-9826-1568d98471b4" containerName="istio-proxy" containerID="cri-o://0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0" gracePeriod=30
Oct 30 21:22:44.583581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.581445    4918 generic.go:356] "Generic (PLEG): container finished" podID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerID="628b293d70552b7591bf94a822b9c14394cc0dba5f8aa6678bd9c9dfd42c1286" exitCode=0
Oct 30 21:22:44.583581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.581497    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" event={"ID":"d919503b-cf6f-4c14-8a83-167f5c35f2cd","Type":"ContainerDied","Data":"628b293d70552b7591bf94a822b9c14394cc0dba5f8aa6678bd9c9dfd42c1286"}
Oct 30 21:22:44.583581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.583123    4918 generic.go:356] "Generic (PLEG): container finished" podID="6521302f-c44e-40f0-b342-10c81833e27a" containerID="8c5cc264f3f94f219169be1bb5546b804acfa72aaaa18905f59fc495f308afc2" exitCode=0
Oct 30 21:22:44.583581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.583192    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" event={"ID":"6521302f-c44e-40f0-b342-10c81833e27a","Type":"ContainerDied","Data":"8c5cc264f3f94f219169be1bb5546b804acfa72aaaa18905f59fc495f308afc2"}
Oct 30 21:22:44.583581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.583235    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj" event={"ID":"6521302f-c44e-40f0-b342-10c81833e27a","Type":"ContainerDied","Data":"a15db8875bd42687ceae9736aa5b90f3f1ada299a8d5ae2562589090a0fd36c7"}
Oct 30 21:22:44.583581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.583250    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a15db8875bd42687ceae9736aa5b90f3f1ada299a8d5ae2562589090a0fd36c7"
Oct 30 21:22:44.588471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.584310    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:22:44.606226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.606203    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:22:44.687124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687091    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-credential-socket\") pod \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") "
Oct 30 21:22:44.687124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687126    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-envoy\") pod \"6521302f-c44e-40f0-b342-10c81833e27a\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") "
Oct 30 21:22:44.687357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687147    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-kube-api-access-rf5hp\" (UniqueName: \"kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-kube-api-access-rf5hp\") pod \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") "
Oct 30 21:22:44.687357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687205    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-certs\") pod \"6521302f-c44e-40f0-b342-10c81833e27a\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") "
Oct 30 21:22:44.687357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687251    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-socket\") pod \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") "
Oct 30 21:22:44.687357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687350    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-certs\") pod \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") "
Oct 30 21:22:44.687742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687476    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istiod-ca-cert\") pod \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") "
Oct 30 21:22:44.687742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687553    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-envoy\") pod \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") "
Oct 30 21:22:44.687742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687551    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-credential-socket" pod "d919503b-cf6f-4c14-8a83-167f5c35f2cd" (UID: "d919503b-cf6f-4c14-8a83-167f5c35f2cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:44.687742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687639    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-socket\") pod \"6521302f-c44e-40f0-b342-10c81833e27a\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") "
Oct 30 21:22:44.687742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687684    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-kube-api-access-qcnqv\" (UniqueName: \"kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-kube-api-access-qcnqv\") pod \"6521302f-c44e-40f0-b342-10c81833e27a\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") "
Oct 30 21:22:44.687742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687714    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-token\" (UniqueName: \"kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-token\") pod \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") "
Oct 30 21:22:44.688040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687750    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6521302f-c44e-40f0-b342-10c81833e27a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6521302f-c44e-40f0-b342-10c81833e27a-istiod-ca-cert\") pod \"6521302f-c44e-40f0-b342-10c81833e27a\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") "
Oct 30 21:22:44.688040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687780    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6521302f-c44e-40f0-b342-10c81833e27a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6521302f-c44e-40f0-b342-10c81833e27a-istio-podinfo\") pod \"6521302f-c44e-40f0-b342-10c81833e27a\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") "
Oct 30 21:22:44.688040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687820    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-istio-token\" (UniqueName: \"kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-istio-token\") pod \"6521302f-c44e-40f0-b342-10c81833e27a\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") "
Oct 30 21:22:44.688040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687855    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-data\") pod \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") "
Oct 30 21:22:44.688040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687857    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-certs" pod "6521302f-c44e-40f0-b342-10c81833e27a" (UID: "6521302f-c44e-40f0-b342-10c81833e27a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:44.688040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687969    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-podinfo\") pod \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\" (UID: \"d919503b-cf6f-4c14-8a83-167f5c35f2cd\") "
Oct 30 21:22:44.688040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.688015    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-credential-socket\") pod \"6521302f-c44e-40f0-b342-10c81833e27a\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") "
Oct 30 21:22:44.688040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.688040    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-data\") pod \"6521302f-c44e-40f0-b342-10c81833e27a\" (UID: \"6521302f-c44e-40f0-b342-10c81833e27a\") "
Oct 30 21:22:44.688768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.687967    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-socket" pod "d919503b-cf6f-4c14-8a83-167f5c35f2cd" (UID: "d919503b-cf6f-4c14-8a83-167f5c35f2cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:44.688768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.688033    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-certs" pod "d919503b-cf6f-4c14-8a83-167f5c35f2cd" (UID: "d919503b-cf6f-4c14-8a83-167f5c35f2cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:44.688768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.688457    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-data" pod "d919503b-cf6f-4c14-8a83-167f5c35f2cd" (UID: "d919503b-cf6f-4c14-8a83-167f5c35f2cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:44.688768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.688642    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-socket" pod "6521302f-c44e-40f0-b342-10c81833e27a" (UID: "6521302f-c44e-40f0-b342-10c81833e27a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:44.688768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.688676    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istiod-ca-cert" pod "d919503b-cf6f-4c14-8a83-167f5c35f2cd" (UID: "d919503b-cf6f-4c14-8a83-167f5c35f2cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:44.689084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.688946    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-credential-socket" pod "6521302f-c44e-40f0-b342-10c81833e27a" (UID: "6521302f-c44e-40f0-b342-10c81833e27a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:44.689167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.689146    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-data" pod "6521302f-c44e-40f0-b342-10c81833e27a" (UID: "6521302f-c44e-40f0-b342-10c81833e27a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:44.689167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.689156    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6521302f-c44e-40f0-b342-10c81833e27a-istiod-ca-cert" pod "6521302f-c44e-40f0-b342-10c81833e27a" (UID: "6521302f-c44e-40f0-b342-10c81833e27a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:44.689493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.689468    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.689612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.689496    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6521302f-c44e-40f0-b342-10c81833e27a-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.689612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.689510    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.689612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.689524    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.689612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.689538    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.689612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.689552    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.689612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.689565    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.689612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.689580    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.689612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.689592    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.689612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.689603    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.691314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.691293    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-envoy" pod "6521302f-c44e-40f0-b342-10c81833e27a" (UID: "6521302f-c44e-40f0-b342-10c81833e27a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:44.691435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.691337    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-token" pod "d919503b-cf6f-4c14-8a83-167f5c35f2cd" (UID: "d919503b-cf6f-4c14-8a83-167f5c35f2cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:44.691816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.691794    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-podinfo" pod "d919503b-cf6f-4c14-8a83-167f5c35f2cd" (UID: "d919503b-cf6f-4c14-8a83-167f5c35f2cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:44.692189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.692167    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-kube-api-access-rf5hp" pod "d919503b-cf6f-4c14-8a83-167f5c35f2cd" (UID: "d919503b-cf6f-4c14-8a83-167f5c35f2cd"). InnerVolumeSpecName "kube-api-access-rf5hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:44.692360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.692338    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-istio-token" pod "6521302f-c44e-40f0-b342-10c81833e27a" (UID: "6521302f-c44e-40f0-b342-10c81833e27a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:44.692456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.692340    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-envoy" pod "d919503b-cf6f-4c14-8a83-167f5c35f2cd" (UID: "d919503b-cf6f-4c14-8a83-167f5c35f2cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:44.693461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.693376    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-kube-api-access-qcnqv" pod "6521302f-c44e-40f0-b342-10c81833e27a" (UID: "6521302f-c44e-40f0-b342-10c81833e27a"). InnerVolumeSpecName "kube-api-access-qcnqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:44.693539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.693523    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6521302f-c44e-40f0-b342-10c81833e27a-istio-podinfo" pod "6521302f-c44e-40f0-b342-10c81833e27a" (UID: "6521302f-c44e-40f0-b342-10c81833e27a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:44.790705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.790663    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.790705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.790695    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6521302f-c44e-40f0-b342-10c81833e27a-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.790705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.790709    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf5hp\" (UniqueName: \"kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-kube-api-access-rf5hp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.790965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.790723    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.790965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.790735    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcnqv\" (UniqueName: \"kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-kube-api-access-qcnqv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.790965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.790748    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d919503b-cf6f-4c14-8a83-167f5c35f2cd-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.790965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.790760    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6521302f-c44e-40f0-b342-10c81833e27a-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:44.790965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:44.790771    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6521302f-c44e-40f0-b342-10c81833e27a-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:45.592174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.592139    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"
Oct 30 21:22:45.592829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.592145    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx" event={"ID":"d919503b-cf6f-4c14-8a83-167f5c35f2cd","Type":"ContainerDied","Data":"30c7ce2d3026057cfd46eac6cb2bb2376a2787418c762025de404f0ee8b21a21"}
Oct 30 21:22:45.592829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.592301    4918 scope.go:122] "RemoveContainer" containerID="628b293d70552b7591bf94a822b9c14394cc0dba5f8aa6678bd9c9dfd42c1286"
Oct 30 21:22:45.592829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.592433    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"
Oct 30 21:22:45.616270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.616246    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"]
Oct 30 21:22:45.620746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.620724    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-hpfvj"]
Oct 30 21:22:45.633455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.631446    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"]
Oct 30 21:22:45.635222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.635205    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-gm9cx"]
Oct 30 21:22:45.935277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.935198    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"]
Oct 30 21:22:45.935858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.935845    4918 cpu_manager.go:414] "containerMap: removing container" podUID="0985b700-2e5c-4adc-84f9-3102272a76ad" containerName="payload-pre-processing"
Oct 30 21:22:45.935918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.935860    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0985b700-2e5c-4adc-84f9-3102272a76ad" containerName="payload-pre-processing"
Oct 30 21:22:45.935918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.935874    4918 cpu_manager.go:414] "containerMap: removing container" podUID="7da1b1bd-9254-4036-b851-d814bbfbaaf3" containerName="revoke-keys"
Oct 30 21:22:45.935918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.935880    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7da1b1bd-9254-4036-b851-d814bbfbaaf3" containerName="revoke-keys"
Oct 30 21:22:45.935918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.935904    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4094667e-929d-45e3-a0ca-4167b3895ca3" containerName="payload-processing"
Oct 30 21:22:45.935918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.935909    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4094667e-929d-45e3-a0ca-4167b3895ca3" containerName="payload-processing"
Oct 30 21:22:45.935918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.935917    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy"
Oct 30 21:22:45.936106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.935922    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy"
Oct 30 21:22:45.936106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.935935    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6521302f-c44e-40f0-b342-10c81833e27a" containerName="istio-proxy"
Oct 30 21:22:45.936106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.935939    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6521302f-c44e-40f0-b342-10c81833e27a" containerName="istio-proxy"
Oct 30 21:22:45.936106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.936047    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6521302f-c44e-40f0-b342-10c81833e27a" containerName="istio-proxy"
Oct 30 21:22:45.936106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.936060    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0985b700-2e5c-4adc-84f9-3102272a76ad" containerName="payload-pre-processing"
Oct 30 21:22:45.936106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.936067    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7da1b1bd-9254-4036-b851-d814bbfbaaf3" containerName="revoke-keys"
Oct 30 21:22:45.936106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.936076    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4094667e-929d-45e3-a0ca-4167b3895ca3" containerName="payload-processing"
Oct 30 21:22:45.936106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.936081    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" containerName="istio-proxy"
Oct 30 21:22:45.940717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.940692    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:45.946483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.946456    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:22:45.946483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.946473    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:22:45.946741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.946542    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:22:45.946741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.946544    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:22:45.947905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:45.947881    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"]
Oct 30 21:22:46.027612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.027574    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smj8\" (UniqueName: \"kubernetes.io/projected/4749573e-75c5-4d09-b8fd-4800afd463eb-kube-api-access-8smj8\") pod \"maas-api-revoke-sub-ed89f361-2vc9z\" (UID: \"4749573e-75c5-4d09-b8fd-4800afd463eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:46.027796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.027634    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed89f361-2vc9z\" (UID: \"4749573e-75c5-4d09-b8fd-4800afd463eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:46.027840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.027801    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed89f361-2vc9z\" (UID: \"4749573e-75c5-4d09-b8fd-4800afd463eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:46.129569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.129540    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smj8\" (UniqueName: \"kubernetes.io/projected/4749573e-75c5-4d09-b8fd-4800afd463eb-kube-api-access-8smj8\") pod \"maas-api-revoke-sub-ed89f361-2vc9z\" (UID: \"4749573e-75c5-4d09-b8fd-4800afd463eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:46.129758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.129587    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed89f361-2vc9z\" (UID: \"4749573e-75c5-4d09-b8fd-4800afd463eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:46.129758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.129730    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed89f361-2vc9z\" (UID: \"4749573e-75c5-4d09-b8fd-4800afd463eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:46.130296 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.130271    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed89f361-2vc9z\" (UID: \"4749573e-75c5-4d09-b8fd-4800afd463eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:46.132300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.132276    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed89f361-2vc9z\" (UID: \"4749573e-75c5-4d09-b8fd-4800afd463eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:46.138287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.138267    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smj8\" (UniqueName: \"kubernetes.io/projected/4749573e-75c5-4d09-b8fd-4800afd463eb-kube-api-access-8smj8\") pod \"maas-api-revoke-sub-ed89f361-2vc9z\" (UID: \"4749573e-75c5-4d09-b8fd-4800afd463eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:46.256487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.255219    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"]
Oct 30 21:22:46.256487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.255503    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx" podUID="32638ae5-1e74-47de-b9ed-2c04e3811409" containerName="istio-proxy" containerID="cri-o://5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f" gracePeriod=30
Oct 30 21:22:46.273294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.261267    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"]
Oct 30 21:22:46.273294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.261332    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:46.273294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.261501    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" podUID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" containerName="istio-proxy" containerID="cri-o://424bdf4cd0bd06d47e64d4edc0dbe55f1a175fb3569d820f4478dd9bf5c78585" gracePeriod=30
Oct 30 21:22:46.296689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.296662    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:22:46.296837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.296713    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" podUID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:22:46.555640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.555614    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"]
Oct 30 21:22:46.556682 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:22:46.556657    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4749573e_75c5_4d09_b8fd_4800afd463eb.slice/crio-737e7ae7177b7a828f316dacfe956cd69186ec407ecca370517b252e55816e3d WatchSource:0}: Error finding container 737e7ae7177b7a828f316dacfe956cd69186ec407ecca370517b252e55816e3d: Status 404 returned error can't find the container with id 737e7ae7177b7a828f316dacfe956cd69186ec407ecca370517b252e55816e3d
Oct 30 21:22:46.600460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:46.600429    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z" event={"ID":"4749573e-75c5-4d09-b8fd-4800afd463eb","Type":"ContainerStarted","Data":"737e7ae7177b7a828f316dacfe956cd69186ec407ecca370517b252e55816e3d"}
Oct 30 21:22:46.795352 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:46.795324    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:47.079641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:47.079609    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6521302f-c44e-40f0-b342-10c81833e27a" path="/var/lib/kubelet/pods/6521302f-c44e-40f0-b342-10c81833e27a/volumes"
Oct 30 21:22:47.080118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:47.080105    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d919503b-cf6f-4c14-8a83-167f5c35f2cd" path="/var/lib/kubelet/pods/d919503b-cf6f-4c14-8a83-167f5c35f2cd/volumes"
Oct 30 21:22:47.611767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:47.611737    4918 generic.go:356] "Generic (PLEG): container finished" podID="4749573e-75c5-4d09-b8fd-4800afd463eb" containerID="143cef77a95d989c40081c47c5cbda1c5d9dd7e1f9521965fa4fb5e97b7cfddb" exitCode=0
Oct 30 21:22:47.612169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:47.611821    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z" event={"ID":"4749573e-75c5-4d09-b8fd-4800afd463eb","Type":"ContainerDied","Data":"143cef77a95d989c40081c47c5cbda1c5d9dd7e1f9521965fa4fb5e97b7cfddb"}
Oct 30 21:22:48.960630 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:48.959464    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:49.009937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.009915    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:49.040043 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.040012    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-xks-ca\") pod \"4749573e-75c5-4d09-b8fd-4800afd463eb\" (UID: \"4749573e-75c5-4d09-b8fd-4800afd463eb\") "
Oct 30 21:22:49.040223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.040139    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4749573e-75c5-4d09-b8fd-4800afd463eb-kube-api-access-8smj8\" (UniqueName: \"kubernetes.io/projected/4749573e-75c5-4d09-b8fd-4800afd463eb-kube-api-access-8smj8\") pod \"4749573e-75c5-4d09-b8fd-4800afd463eb\" (UID: \"4749573e-75c5-4d09-b8fd-4800afd463eb\") "
Oct 30 21:22:49.040305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.040249    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-openshift-ca\") pod \"4749573e-75c5-4d09-b8fd-4800afd463eb\" (UID: \"4749573e-75c5-4d09-b8fd-4800afd463eb\") "
Oct 30 21:22:49.040686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.040625    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-openshift-ca" pod "4749573e-75c5-4d09-b8fd-4800afd463eb" (UID: "4749573e-75c5-4d09-b8fd-4800afd463eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:49.041152 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.041132    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:49.042594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.042572    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4749573e-75c5-4d09-b8fd-4800afd463eb-kube-api-access-8smj8" pod "4749573e-75c5-4d09-b8fd-4800afd463eb" (UID: "4749573e-75c5-4d09-b8fd-4800afd463eb"). InnerVolumeSpecName "kube-api-access-8smj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:49.042772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.042754    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-xks-ca" pod "4749573e-75c5-4d09-b8fd-4800afd463eb" (UID: "4749573e-75c5-4d09-b8fd-4800afd463eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:49.054947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.054919    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"]
Oct 30 21:22:49.083985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.083940    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"]
Oct 30 21:22:49.152409 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.152361    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8smj8\" (UniqueName: \"kubernetes.io/projected/4749573e-75c5-4d09-b8fd-4800afd463eb-kube-api-access-8smj8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:49.152585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.152424    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4749573e-75c5-4d09-b8fd-4800afd463eb-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:49.191711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.191669    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:22:49.196535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.196507    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"]
Oct 30 21:22:49.197218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.197198    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4749573e-75c5-4d09-b8fd-4800afd463eb" containerName="revoke-keys"
Oct 30 21:22:49.197329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.197219    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4749573e-75c5-4d09-b8fd-4800afd463eb" containerName="revoke-keys"
Oct 30 21:22:49.197329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.197243    4918 cpu_manager.go:414] "containerMap: removing container" podUID="0a415d16-9fc3-4b38-9826-1568d98471b4" containerName="istio-proxy"
Oct 30 21:22:49.197329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.197251    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a415d16-9fc3-4b38-9826-1568d98471b4" containerName="istio-proxy"
Oct 30 21:22:49.197565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.197551    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4749573e-75c5-4d09-b8fd-4800afd463eb" containerName="revoke-keys"
Oct 30 21:22:49.197620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.197572    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a415d16-9fc3-4b38-9826-1568d98471b4" containerName="istio-proxy"
Oct 30 21:22:49.233866 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.233835    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"]
Oct 30 21:22:49.234038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.233958    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:49.381414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.379502    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-envoy\") pod \"0a415d16-9fc3-4b38-9826-1568d98471b4\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") "
Oct 30 21:22:49.381414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.379596    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-token\" (UniqueName: \"kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-token\") pod \"0a415d16-9fc3-4b38-9826-1568d98471b4\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") "
Oct 30 21:22:49.381414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.379664    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-data\") pod \"0a415d16-9fc3-4b38-9826-1568d98471b4\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") "
Oct 30 21:22:49.381414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.379731    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-socket\") pod \"0a415d16-9fc3-4b38-9826-1568d98471b4\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") "
Oct 30 21:22:49.381414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.379790    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-kube-api-access-2czgs\" (UniqueName: \"kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-kube-api-access-2czgs\") pod \"0a415d16-9fc3-4b38-9826-1568d98471b4\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") "
Oct 30 21:22:49.381414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.379962    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-credential-socket\") pod \"0a415d16-9fc3-4b38-9826-1568d98471b4\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") "
Oct 30 21:22:49.381414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.380020    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-podinfo\") pod \"0a415d16-9fc3-4b38-9826-1568d98471b4\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") "
Oct 30 21:22:49.381414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.380056    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-certs\") pod \"0a415d16-9fc3-4b38-9826-1568d98471b4\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") "
Oct 30 21:22:49.381414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.380122    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a415d16-9fc3-4b38-9826-1568d98471b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a415d16-9fc3-4b38-9826-1568d98471b4-istiod-ca-cert\") pod \"0a415d16-9fc3-4b38-9826-1568d98471b4\" (UID: \"0a415d16-9fc3-4b38-9826-1568d98471b4\") "
Oct 30 21:22:49.381414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.380449    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74q6d\" (UniqueName: \"kubernetes.io/projected/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-kube-api-access-74q6d\") pod \"maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd\" (UID: \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:49.381414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.380588    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd\" (UID: \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:49.381414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.380718    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd\" (UID: \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:49.387973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.387491    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-socket" pod "0a415d16-9fc3-4b38-9826-1568d98471b4" (UID: "0a415d16-9fc3-4b38-9826-1568d98471b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:49.387973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.387718    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-certs" pod "0a415d16-9fc3-4b38-9826-1568d98471b4" (UID: "0a415d16-9fc3-4b38-9826-1568d98471b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:49.412733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.390466    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-credential-socket" pod "0a415d16-9fc3-4b38-9826-1568d98471b4" (UID: "0a415d16-9fc3-4b38-9826-1568d98471b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:49.413041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.413009    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-data" pod "0a415d16-9fc3-4b38-9826-1568d98471b4" (UID: "0a415d16-9fc3-4b38-9826-1568d98471b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:49.413360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.413286    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a415d16-9fc3-4b38-9826-1568d98471b4-istiod-ca-cert" pod "0a415d16-9fc3-4b38-9826-1568d98471b4" (UID: "0a415d16-9fc3-4b38-9826-1568d98471b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:49.424769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.424701    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-token" pod "0a415d16-9fc3-4b38-9826-1568d98471b4" (UID: "0a415d16-9fc3-4b38-9826-1568d98471b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:49.436415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.428842    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-envoy" pod "0a415d16-9fc3-4b38-9826-1568d98471b4" (UID: "0a415d16-9fc3-4b38-9826-1568d98471b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:49.436415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.429624    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-kube-api-access-2czgs" pod "0a415d16-9fc3-4b38-9826-1568d98471b4" (UID: "0a415d16-9fc3-4b38-9826-1568d98471b4"). InnerVolumeSpecName "kube-api-access-2czgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:49.436415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.429739    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-podinfo" pod "0a415d16-9fc3-4b38-9826-1568d98471b4" (UID: "0a415d16-9fc3-4b38-9826-1568d98471b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.482062    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74q6d\" (UniqueName: \"kubernetes.io/projected/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-kube-api-access-74q6d\") pod \"maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd\" (UID: \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.482185    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd\" (UID: \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.482256    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd\" (UID: \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.482363    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.482381    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.482415    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.482429    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a415d16-9fc3-4b38-9826-1568d98471b4-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.482442    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.482456    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.482468    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.482480    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a415d16-9fc3-4b38-9826-1568d98471b4-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.482494    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2czgs\" (UniqueName: \"kubernetes.io/projected/0a415d16-9fc3-4b38-9826-1568d98471b4-kube-api-access-2czgs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:49.486484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.483287    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd\" (UID: \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:49.491414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.490459    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd\" (UID: \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:49.518821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.518781    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74q6d\" (UniqueName: \"kubernetes.io/projected/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-kube-api-access-74q6d\") pod \"maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd\" (UID: \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:49.554814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.554461    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:49.651554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.650994    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z" event={"ID":"4749573e-75c5-4d09-b8fd-4800afd463eb","Type":"ContainerDied","Data":"737e7ae7177b7a828f316dacfe956cd69186ec407ecca370517b252e55816e3d"}
Oct 30 21:22:49.651554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.651041    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"
Oct 30 21:22:49.651554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.651075    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="737e7ae7177b7a828f316dacfe956cd69186ec407ecca370517b252e55816e3d"
Oct 30 21:22:49.655529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.655497    4918 generic.go:356] "Generic (PLEG): container finished" podID="0a415d16-9fc3-4b38-9826-1568d98471b4" containerID="0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0" exitCode=0
Oct 30 21:22:49.655646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.655603    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k" event={"ID":"0a415d16-9fc3-4b38-9826-1568d98471b4","Type":"ContainerDied","Data":"0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0"}
Oct 30 21:22:49.655727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.655646    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k" event={"ID":"0a415d16-9fc3-4b38-9826-1568d98471b4","Type":"ContainerDied","Data":"af659c7ade804546c177a77dd918f34e3c5de8cea21686b319122c0a90257c19"}
Oct 30 21:22:49.655727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.655677    4918 scope.go:122] "RemoveContainer" containerID="0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0"
Oct 30 21:22:49.655845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.655815    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"
Oct 30 21:22:49.697717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.697698    4918 scope.go:122] "RemoveContainer" containerID="0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0"
Oct 30 21:22:49.698095 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:49.698075    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0\": container with ID starting with 0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0 not found: ID does not exist" containerID="0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0"
Oct 30 21:22:49.698187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.698106    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0"} err="failed to get container status \"0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0\": rpc error: code = NotFound desc = could not find container \"0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0\": container with ID starting with 0c0fdf7f9c1cc4b4668666d17e03a201d40b6ac132a38d33c5ca3377e08e9bd0 not found: ID does not exist"
Oct 30 21:22:49.712252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.712230    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"]
Oct 30 21:22:49.724259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.724237    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-m8s6k"]
Oct 30 21:22:49.854407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:49.854370    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"]
Oct 30 21:22:49.855140 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:22:49.855115    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c8b17c_8a8b_43e0_abbd_5eb5a8b6b167.slice/crio-5285313d5f6a51789b9ea086769c03be150d47e1e44a8d87bdf4042dae2f000c WatchSource:0}: Error finding container 5285313d5f6a51789b9ea086769c03be150d47e1e44a8d87bdf4042dae2f000c: Status 404 returned error can't find the container with id 5285313d5f6a51789b9ea086769c03be150d47e1e44a8d87bdf4042dae2f000c
Oct 30 21:22:50.666423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:50.666310    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd" event={"ID":"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167","Type":"ContainerStarted","Data":"034579598db6d3c8827222da9c7475df0547efe9b9bc668e46145536ce1ee248"}
Oct 30 21:22:50.666895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:50.666434    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd" event={"ID":"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167","Type":"ContainerStarted","Data":"5285313d5f6a51789b9ea086769c03be150d47e1e44a8d87bdf4042dae2f000c"}
Oct 30 21:22:50.690983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:50.690929    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd" podStartSLOduration=1.6909150689999999 podStartE2EDuration="1.690915069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:50.687855876 +0000 UTC m=+1283.734415045" watchObservedRunningTime="2026-10-30 21:22:50.690915069 +0000 UTC m=+1283.737474246"
Oct 30 21:22:51.079526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.079496    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a415d16-9fc3-4b38-9826-1568d98471b4" path="/var/lib/kubelet/pods/0a415d16-9fc3-4b38-9826-1568d98471b4/volumes"
Oct 30 21:22:51.709471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.709426    4918 generic.go:356] "Generic (PLEG): container finished" podID="75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167" containerID="034579598db6d3c8827222da9c7475df0547efe9b9bc668e46145536ce1ee248" exitCode=0
Oct 30 21:22:51.709887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.709798    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd" event={"ID":"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167","Type":"ContainerDied","Data":"034579598db6d3c8827222da9c7475df0547efe9b9bc668e46145536ce1ee248"}
Oct 30 21:22:51.710817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.710752    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:22:51.711545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.711522    4918 generic.go:356] "Generic (PLEG): container finished" podID="32638ae5-1e74-47de-b9ed-2c04e3811409" containerID="5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f" exitCode=0
Oct 30 21:22:51.711667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.711594    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx" event={"ID":"32638ae5-1e74-47de-b9ed-2c04e3811409","Type":"ContainerDied","Data":"5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f"}
Oct 30 21:22:51.711667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.711632    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx" event={"ID":"32638ae5-1e74-47de-b9ed-2c04e3811409","Type":"ContainerDied","Data":"225e4adc2ac7f8194a1c17ebb7279c8cad8681232e4fd53b6550456ee9c872e2"}
Oct 30 21:22:51.711667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.711653    4918 scope.go:122] "RemoveContainer" containerID="5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f"
Oct 30 21:22:51.715856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.715468    4918 generic.go:356] "Generic (PLEG): container finished" podID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" containerID="424bdf4cd0bd06d47e64d4edc0dbe55f1a175fb3569d820f4478dd9bf5c78585" exitCode=0
Oct 30 21:22:51.715856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.715630    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" event={"ID":"c40df71b-5ccf-4655-bee6-eec8cf48d47b","Type":"ContainerDied","Data":"424bdf4cd0bd06d47e64d4edc0dbe55f1a175fb3569d820f4478dd9bf5c78585"}
Oct 30 21:22:51.734066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.734041    4918 scope.go:122] "RemoveContainer" containerID="5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f"
Oct 30 21:22:51.734456 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:51.734428    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f\": container with ID starting with 5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f not found: ID does not exist" containerID="5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f"
Oct 30 21:22:51.734652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.734614    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f"} err="failed to get container status \"5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f\": rpc error: code = NotFound desc = could not find container \"5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f\": container with ID starting with 5233eeda2867589ccb8faf077818e1609d5951e8ddbcc163590407ae90cb948f not found: ID does not exist"
Oct 30 21:22:51.748175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.748144    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:22:51.833236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833206    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-credential-socket\") pod \"32638ae5-1e74-47de-b9ed-2c04e3811409\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") "
Oct 30 21:22:51.833425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833275    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32638ae5-1e74-47de-b9ed-2c04e3811409-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32638ae5-1e74-47de-b9ed-2c04e3811409-istiod-ca-cert\") pod \"32638ae5-1e74-47de-b9ed-2c04e3811409\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") "
Oct 30 21:22:51.833425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833381    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-certs\") pod \"32638ae5-1e74-47de-b9ed-2c04e3811409\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") "
Oct 30 21:22:51.833564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833436    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-envoy\") pod \"32638ae5-1e74-47de-b9ed-2c04e3811409\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") "
Oct 30 21:22:51.833625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833560    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-token\" (UniqueName: \"kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-token\") pod \"32638ae5-1e74-47de-b9ed-2c04e3811409\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") "
Oct 30 21:22:51.833625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833600    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-credential-socket" pod "32638ae5-1e74-47de-b9ed-2c04e3811409" (UID: "32638ae5-1e74-47de-b9ed-2c04e3811409"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.833734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833630    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-certs" pod "32638ae5-1e74-47de-b9ed-2c04e3811409" (UID: "32638ae5-1e74-47de-b9ed-2c04e3811409"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.833734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833659    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-socket\") pod \"32638ae5-1e74-47de-b9ed-2c04e3811409\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") "
Oct 30 21:22:51.833734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833694    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-data\") pod \"32638ae5-1e74-47de-b9ed-2c04e3811409\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") "
Oct 30 21:22:51.833734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833724    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-kube-api-access-pbhp9\" (UniqueName: \"kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-kube-api-access-pbhp9\") pod \"32638ae5-1e74-47de-b9ed-2c04e3811409\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") "
Oct 30 21:22:51.833734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833722    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32638ae5-1e74-47de-b9ed-2c04e3811409-istiod-ca-cert" pod "32638ae5-1e74-47de-b9ed-2c04e3811409" (UID: "32638ae5-1e74-47de-b9ed-2c04e3811409"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:51.833977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833847    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-podinfo\") pod \"32638ae5-1e74-47de-b9ed-2c04e3811409\" (UID: \"32638ae5-1e74-47de-b9ed-2c04e3811409\") "
Oct 30 21:22:51.833977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.833950    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-socket" pod "32638ae5-1e74-47de-b9ed-2c04e3811409" (UID: "32638ae5-1e74-47de-b9ed-2c04e3811409"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.834070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.834043    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-data" pod "32638ae5-1e74-47de-b9ed-2c04e3811409" (UID: "32638ae5-1e74-47de-b9ed-2c04e3811409"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.834627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.834561    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:51.834627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.834591    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:51.834627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.834606    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:51.834627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.834620    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:51.834956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.834634    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32638ae5-1e74-47de-b9ed-2c04e3811409-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:51.836699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.836675    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-kube-api-access-pbhp9" pod "32638ae5-1e74-47de-b9ed-2c04e3811409" (UID: "32638ae5-1e74-47de-b9ed-2c04e3811409"). InnerVolumeSpecName "kube-api-access-pbhp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:51.836921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.836899    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-token" pod "32638ae5-1e74-47de-b9ed-2c04e3811409" (UID: "32638ae5-1e74-47de-b9ed-2c04e3811409"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:51.837015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.836916    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-envoy" pod "32638ae5-1e74-47de-b9ed-2c04e3811409" (UID: "32638ae5-1e74-47de-b9ed-2c04e3811409"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.837215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.837189    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-podinfo" pod "32638ae5-1e74-47de-b9ed-2c04e3811409" (UID: "32638ae5-1e74-47de-b9ed-2c04e3811409"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.951950    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-certs\") pod \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") "
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.952059    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-token\" (UniqueName: \"kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-token\") pod \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") "
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.952122    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-envoy\") pod \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") "
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.952217    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istiod-ca-cert\") pod \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") "
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.952271    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-kube-api-access-tv68c\" (UniqueName: \"kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-kube-api-access-tv68c\") pod \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") "
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.952347    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-data\") pod \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") "
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.952552    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-credential-socket\") pod \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") "
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.952601    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-socket\") pod \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") "
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.952639    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-podinfo\") pod \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\" (UID: \"c40df71b-5ccf-4655-bee6-eec8cf48d47b\") "
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.953367    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.953459    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.953500    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbhp9\" (UniqueName: \"kubernetes.io/projected/32638ae5-1e74-47de-b9ed-2c04e3811409-kube-api-access-pbhp9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.953514    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32638ae5-1e74-47de-b9ed-2c04e3811409-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.954802    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istiod-ca-cert" pod "c40df71b-5ccf-4655-bee6-eec8cf48d47b" (UID: "c40df71b-5ccf-4655-bee6-eec8cf48d47b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:51.955429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.955102    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-certs" pod "c40df71b-5ccf-4655-bee6-eec8cf48d47b" (UID: "c40df71b-5ccf-4655-bee6-eec8cf48d47b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.958446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.956729    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-podinfo" pod "c40df71b-5ccf-4655-bee6-eec8cf48d47b" (UID: "c40df71b-5ccf-4655-bee6-eec8cf48d47b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:22:51.959049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.958859    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-credential-socket" pod "c40df71b-5ccf-4655-bee6-eec8cf48d47b" (UID: "c40df71b-5ccf-4655-bee6-eec8cf48d47b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.959432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.959240    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-data" pod "c40df71b-5ccf-4655-bee6-eec8cf48d47b" (UID: "c40df71b-5ccf-4655-bee6-eec8cf48d47b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.959650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.959605    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-socket" pod "c40df71b-5ccf-4655-bee6-eec8cf48d47b" (UID: "c40df71b-5ccf-4655-bee6-eec8cf48d47b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.960847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.960803    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-token" pod "c40df71b-5ccf-4655-bee6-eec8cf48d47b" (UID: "c40df71b-5ccf-4655-bee6-eec8cf48d47b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:51.967589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.966885    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-envoy" pod "c40df71b-5ccf-4655-bee6-eec8cf48d47b" (UID: "c40df71b-5ccf-4655-bee6-eec8cf48d47b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:22:51.988944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:51.988890    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-kube-api-access-tv68c" pod "c40df71b-5ccf-4655-bee6-eec8cf48d47b" (UID: "c40df71b-5ccf-4655-bee6-eec8cf48d47b"). InnerVolumeSpecName "kube-api-access-tv68c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:52.054956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.054923    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:52.054956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.054951    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:52.054956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.054962    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv68c\" (UniqueName: \"kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-kube-api-access-tv68c\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:52.055181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.054972    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:52.055181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.054981    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:52.055181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.054990    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:52.055181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.054998    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:52.055181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.055008    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40df71b-5ccf-4655-bee6-eec8cf48d47b-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:52.055181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.055015    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40df71b-5ccf-4655-bee6-eec8cf48d47b-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:52.724779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.724748    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"
Oct 30 21:22:52.726537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.726515    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"
Oct 30 21:22:52.726537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.726510    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc" event={"ID":"c40df71b-5ccf-4655-bee6-eec8cf48d47b","Type":"ContainerDied","Data":"0c009014a650a262dd529cbf450faa3af5dbd4e826f04a29bcfcb6d5124f8635"}
Oct 30 21:22:52.726742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.726578    4918 scope.go:122] "RemoveContainer" containerID="424bdf4cd0bd06d47e64d4edc0dbe55f1a175fb3569d820f4478dd9bf5c78585"
Oct 30 21:22:52.761101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.761058    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"]
Oct 30 21:22:52.766796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.766770    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-t62fx"]
Oct 30 21:22:52.776089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.776058    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"]
Oct 30 21:22:52.780644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.780622    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-2zfjc"]
Oct 30 21:22:52.958970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:52.958944    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:53.071083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.071004    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-xks-ca\") pod \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\" (UID: \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\") "
Oct 30 21:22:53.071083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.071073    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-kube-api-access-74q6d\" (UniqueName: \"kubernetes.io/projected/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-kube-api-access-74q6d\") pod \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\" (UID: \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\") "
Oct 30 21:22:53.071309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.071151    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-openshift-ca\") pod \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\" (UID: \"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167\") "
Oct 30 21:22:53.071586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.071555    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-openshift-ca" pod "75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167" (UID: "75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:53.071728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.071692    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:53.073376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.073348    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-xks-ca" pod "75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167" (UID: "75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:22:53.073553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.073531    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-kube-api-access-74q6d" pod "75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167" (UID: "75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167"). InnerVolumeSpecName "kube-api-access-74q6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:53.078815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.078794    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32638ae5-1e74-47de-b9ed-2c04e3811409" path="/var/lib/kubelet/pods/32638ae5-1e74-47de-b9ed-2c04e3811409/volumes"
Oct 30 21:22:53.079286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.079272    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" path="/var/lib/kubelet/pods/c40df71b-5ccf-4655-bee6-eec8cf48d47b/volumes"
Oct 30 21:22:53.173334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.173303    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:53.173334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.173335    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74q6d\" (UniqueName: \"kubernetes.io/projected/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167-kube-api-access-74q6d\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:53.746293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.746261    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"
Oct 30 21:22:53.746756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.746257    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd" event={"ID":"75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167","Type":"ContainerDied","Data":"5285313d5f6a51789b9ea086769c03be150d47e1e44a8d87bdf4042dae2f000c"}
Oct 30 21:22:53.746756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:53.746485    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5285313d5f6a51789b9ea086769c03be150d47e1e44a8d87bdf4042dae2f000c"
Oct 30 21:22:54.056208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.056106    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69" podUID="e3fe60da-e4fe-46f8-9006-d8097a0a3ef6" containerName="payload-processing" containerID="cri-o://d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2" gracePeriod=25
Oct 30 21:22:54.090020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.089977    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2" podUID="27df2c10-4316-489e-9a7a-19417b131519" containerName="payload-pre-processing" containerID="cri-o://85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9" gracePeriod=25
Oct 30 21:22:54.433881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.432914    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"
Oct 30 21:22:54.500319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.500292    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"
Oct 30 21:22:54.522128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.522098    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-kube-api-access-lkddh\" (UniqueName: \"kubernetes.io/projected/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-kube-api-access-lkddh\") pod \"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6\" (UID: \"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6\") "
Oct 30 21:22:54.522309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.522141    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27df2c10-4316-489e-9a7a-19417b131519-kube-api-access-pqq4t\" (UniqueName: \"kubernetes.io/projected/27df2c10-4316-489e-9a7a-19417b131519-kube-api-access-pqq4t\") pod \"27df2c10-4316-489e-9a7a-19417b131519\" (UID: \"27df2c10-4316-489e-9a7a-19417b131519\") "
Oct 30 21:22:54.522309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.522220    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-plugins-config-volume\") pod \"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6\" (UID: \"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6\") "
Oct 30 21:22:54.522453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.522347    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27df2c10-4316-489e-9a7a-19417b131519-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27df2c10-4316-489e-9a7a-19417b131519-plugins-config-volume\") pod \"27df2c10-4316-489e-9a7a-19417b131519\" (UID: \"27df2c10-4316-489e-9a7a-19417b131519\") "
Oct 30 21:22:54.523048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.522880    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-plugins-config-volume" pod "e3fe60da-e4fe-46f8-9006-d8097a0a3ef6" (UID: "e3fe60da-e4fe-46f8-9006-d8097a0a3ef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:54.523048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.523001    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.523299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.523267    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27df2c10-4316-489e-9a7a-19417b131519-plugins-config-volume" pod "27df2c10-4316-489e-9a7a-19417b131519" (UID: "27df2c10-4316-489e-9a7a-19417b131519"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:54.525823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.525800    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-kube-api-access-lkddh" pod "e3fe60da-e4fe-46f8-9006-d8097a0a3ef6" (UID: "e3fe60da-e4fe-46f8-9006-d8097a0a3ef6"). InnerVolumeSpecName "kube-api-access-lkddh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:54.525920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.525832    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27df2c10-4316-489e-9a7a-19417b131519-kube-api-access-pqq4t" pod "27df2c10-4316-489e-9a7a-19417b131519" (UID: "27df2c10-4316-489e-9a7a-19417b131519"). InnerVolumeSpecName "kube-api-access-pqq4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:54.624571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.624496    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkddh\" (UniqueName: \"kubernetes.io/projected/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6-kube-api-access-lkddh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.624571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.624524    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqq4t\" (UniqueName: \"kubernetes.io/projected/27df2c10-4316-489e-9a7a-19417b131519-kube-api-access-pqq4t\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.624571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.624533    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27df2c10-4316-489e-9a7a-19417b131519-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:54.758014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.757983    4918 generic.go:356] "Generic (PLEG): container finished" podID="e3fe60da-e4fe-46f8-9006-d8097a0a3ef6" containerID="d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2" exitCode=0
Oct 30 21:22:54.758493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.758027    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"
Oct 30 21:22:54.758493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.758063    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69" event={"ID":"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6","Type":"ContainerDied","Data":"d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2"}
Oct 30 21:22:54.758493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.758113    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69" event={"ID":"e3fe60da-e4fe-46f8-9006-d8097a0a3ef6","Type":"ContainerDied","Data":"3110fb4680dc583924431d5dd490233250bd1c8eda90d96c56e72a5f5a686633"}
Oct 30 21:22:54.758493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.758134    4918 scope.go:122] "RemoveContainer" containerID="d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2"
Oct 30 21:22:54.759665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.759640    4918 generic.go:356] "Generic (PLEG): container finished" podID="27df2c10-4316-489e-9a7a-19417b131519" containerID="85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9" exitCode=0
Oct 30 21:22:54.759804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.759683    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"
Oct 30 21:22:54.759804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.759713    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2" event={"ID":"27df2c10-4316-489e-9a7a-19417b131519","Type":"ContainerDied","Data":"85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9"}
Oct 30 21:22:54.759804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.759768    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2" event={"ID":"27df2c10-4316-489e-9a7a-19417b131519","Type":"ContainerDied","Data":"257550a2a0e3794b2f9e70020564cc5ed0ff14c9c2821625b631fb7c5c74188a"}
Oct 30 21:22:54.774941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.773347    4918 scope.go:122] "RemoveContainer" containerID="d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2"
Oct 30 21:22:54.774941 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:54.774786    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2\": container with ID starting with d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2 not found: ID does not exist" containerID="d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2"
Oct 30 21:22:54.774941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.774816    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2"} err="failed to get container status \"d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2\": rpc error: code = NotFound desc = could not find container \"d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2\": container with ID starting with d2d990ed6d1c20aa1a3d35bea3f95418dc6c393583ae486e7df7131381263da2 not found: ID does not exist"
Oct 30 21:22:54.774941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.774836    4918 scope.go:122] "RemoveContainer" containerID="85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9"
Oct 30 21:22:54.786235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.785861    4918 scope.go:122] "RemoveContainer" containerID="85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9"
Oct 30 21:22:54.786691 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:54.786672    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9\": container with ID starting with 85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9 not found: ID does not exist" containerID="85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9"
Oct 30 21:22:54.786799 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.786697    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9"} err="failed to get container status \"85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9\": rpc error: code = NotFound desc = could not find container \"85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9\": container with ID starting with 85d77c794b158d172dd489680bdc8fa921370447cd91fed3b5e6038bdc8d37a9 not found: ID does not exist"
Oct 30 21:22:54.786871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.786841    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"]
Oct 30 21:22:54.790639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.790619    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d87b5f-9488b6b45-rdn69"]
Oct 30 21:22:54.800470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.800449    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"]
Oct 30 21:22:54.804945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:54.804926    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d87b5f-6585c785c5-vwmj2"]
Oct 30 21:22:55.122345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:55.122308    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27df2c10-4316-489e-9a7a-19417b131519" path="/var/lib/kubelet/pods/27df2c10-4316-489e-9a7a-19417b131519/volumes"
Oct 30 21:22:55.123350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:55.123319    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3fe60da-e4fe-46f8-9006-d8097a0a3ef6" path="/var/lib/kubelet/pods/e3fe60da-e4fe-46f8-9006-d8097a0a3ef6/volumes"
Oct 30 21:22:57.860616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:57.860580    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"]
Oct 30 21:22:57.861366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:57.861311    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9" podUID="6012c0fa-471a-41ec-ba1a-95be1768af03" containerName="payload-processing" containerID="cri-o://e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91" gracePeriod=30
Oct 30 21:22:57.878740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:57.872098    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"]
Oct 30 21:22:57.878740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:57.872320    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz" podUID="73e680b3-0724-436c-a33f-217385db1bd1" containerName="payload-pre-processing" containerID="cri-o://089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f" gracePeriod=30
Oct 30 21:22:58.360107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.360079    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"
Oct 30 21:22:58.396306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.375884    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"]
Oct 30 21:22:58.409843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.406832    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4c6998-92vtq"]
Oct 30 21:22:58.468082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.468055    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"
Oct 30 21:22:58.510985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.510957    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6012c0fa-471a-41ec-ba1a-95be1768af03-kube-api-access-52f6x\" (UniqueName: \"kubernetes.io/projected/6012c0fa-471a-41ec-ba1a-95be1768af03-kube-api-access-52f6x\") pod \"6012c0fa-471a-41ec-ba1a-95be1768af03\" (UID: \"6012c0fa-471a-41ec-ba1a-95be1768af03\") "
Oct 30 21:22:58.511172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.511007    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e680b3-0724-436c-a33f-217385db1bd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e680b3-0724-436c-a33f-217385db1bd1-plugins-config-volume\") pod \"73e680b3-0724-436c-a33f-217385db1bd1\" (UID: \"73e680b3-0724-436c-a33f-217385db1bd1\") "
Oct 30 21:22:58.511172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.511075    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6012c0fa-471a-41ec-ba1a-95be1768af03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6012c0fa-471a-41ec-ba1a-95be1768af03-plugins-config-volume\") pod \"6012c0fa-471a-41ec-ba1a-95be1768af03\" (UID: \"6012c0fa-471a-41ec-ba1a-95be1768af03\") "
Oct 30 21:22:58.511355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.511264    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e680b3-0724-436c-a33f-217385db1bd1-kube-api-access-s57wp\" (UniqueName: \"kubernetes.io/projected/73e680b3-0724-436c-a33f-217385db1bd1-kube-api-access-s57wp\") pod \"73e680b3-0724-436c-a33f-217385db1bd1\" (UID: \"73e680b3-0724-436c-a33f-217385db1bd1\") "
Oct 30 21:22:58.511541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.511510    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e680b3-0724-436c-a33f-217385db1bd1-plugins-config-volume" pod "73e680b3-0724-436c-a33f-217385db1bd1" (UID: "73e680b3-0724-436c-a33f-217385db1bd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:58.511635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.511551    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6012c0fa-471a-41ec-ba1a-95be1768af03-plugins-config-volume" pod "6012c0fa-471a-41ec-ba1a-95be1768af03" (UID: "6012c0fa-471a-41ec-ba1a-95be1768af03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:22:58.512045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.512018    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e680b3-0724-436c-a33f-217385db1bd1-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:58.512126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.512049    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6012c0fa-471a-41ec-ba1a-95be1768af03-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:58.513540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.513506    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e680b3-0724-436c-a33f-217385db1bd1-kube-api-access-s57wp" pod "73e680b3-0724-436c-a33f-217385db1bd1" (UID: "73e680b3-0724-436c-a33f-217385db1bd1"). InnerVolumeSpecName "kube-api-access-s57wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:58.513696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.513672    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6012c0fa-471a-41ec-ba1a-95be1768af03-kube-api-access-52f6x" pod "6012c0fa-471a-41ec-ba1a-95be1768af03" (UID: "6012c0fa-471a-41ec-ba1a-95be1768af03"). InnerVolumeSpecName "kube-api-access-52f6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:22:58.613223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.613097    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s57wp\" (UniqueName: \"kubernetes.io/projected/73e680b3-0724-436c-a33f-217385db1bd1-kube-api-access-s57wp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:58.613525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.613494    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52f6x\" (UniqueName: \"kubernetes.io/projected/6012c0fa-471a-41ec-ba1a-95be1768af03-kube-api-access-52f6x\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:22:58.798521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.798489    4918 generic.go:356] "Generic (PLEG): container finished" podID="73e680b3-0724-436c-a33f-217385db1bd1" containerID="089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f" exitCode=0
Oct 30 21:22:58.798719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.798541    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"
Oct 30 21:22:58.798719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.798575    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz" event={"ID":"73e680b3-0724-436c-a33f-217385db1bd1","Type":"ContainerDied","Data":"089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f"}
Oct 30 21:22:58.798719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.798637    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz" event={"ID":"73e680b3-0724-436c-a33f-217385db1bd1","Type":"ContainerDied","Data":"aa3a47b7b7871d92b67fee363ccbdbae9a5bb8105b2aa2b9fde0cc7b859517b9"}
Oct 30 21:22:58.798719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.798669    4918 scope.go:122] "RemoveContainer" containerID="089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f"
Oct 30 21:22:58.800624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.800597    4918 generic.go:356] "Generic (PLEG): container finished" podID="6012c0fa-471a-41ec-ba1a-95be1768af03" containerID="e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91" exitCode=0
Oct 30 21:22:58.800740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.800647    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"
Oct 30 21:22:58.800740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.800650    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9" event={"ID":"6012c0fa-471a-41ec-ba1a-95be1768af03","Type":"ContainerDied","Data":"e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91"}
Oct 30 21:22:58.800740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.800691    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9" event={"ID":"6012c0fa-471a-41ec-ba1a-95be1768af03","Type":"ContainerDied","Data":"08577f4579dc9e1f5f374b0e5c4e65ad7e80004a4d325b3b02997cf8fdce2150"}
Oct 30 21:22:58.811408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.811363    4918 scope.go:122] "RemoveContainer" containerID="089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f"
Oct 30 21:22:58.811699 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:58.811676    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f\": container with ID starting with 089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f not found: ID does not exist" containerID="089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f"
Oct 30 21:22:58.811802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.811713    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f"} err="failed to get container status \"089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f\": rpc error: code = NotFound desc = could not find container \"089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f\": container with ID starting with 089821b730b0a1ef9a5f711e4348ad88cbb048a895ee6159a94147919ee8b14f not found: ID does not exist"
Oct 30 21:22:58.811802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.811734    4918 scope.go:122] "RemoveContainer" containerID="e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91"
Oct 30 21:22:58.822319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.822257    4918 scope.go:122] "RemoveContainer" containerID="e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91"
Oct 30 21:22:58.822632 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:58.822608    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91\": container with ID starting with e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91 not found: ID does not exist" containerID="e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91"
Oct 30 21:22:58.822710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.822644    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91"} err="failed to get container status \"e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91\": rpc error: code = NotFound desc = could not find container \"e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91\": container with ID starting with e99e953d799374b827324b9dd7f273aab4362d996d2680077c8c43e454b82c91 not found: ID does not exist"
Oct 30 21:22:58.826040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.826021    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"]
Oct 30 21:22:58.837111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.837092    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c6998-b96b7db7-s9nsz"]
Oct 30 21:22:58.848893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.848869    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"]
Oct 30 21:22:58.854247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:58.854222    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c6998-5b4956ccd8-cr2n9"]
Oct 30 21:22:59.037942 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:59.037906    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f\": RecentStats: unable to find data in memory cache]"
Oct 30 21:22:59.079005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:59.078975    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45addcc7-b265-418c-bbbd-f9a2a275d8c5" path="/var/lib/kubelet/pods/45addcc7-b265-418c-bbbd-f9a2a275d8c5/volumes"
Oct 30 21:22:59.079435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:59.079421    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6012c0fa-471a-41ec-ba1a-95be1768af03" path="/var/lib/kubelet/pods/6012c0fa-471a-41ec-ba1a-95be1768af03/volumes"
Oct 30 21:22:59.079794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:22:59.079782    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e680b3-0724-436c-a33f-217385db1bd1" path="/var/lib/kubelet/pods/73e680b3-0724-436c-a33f-217385db1bd1/volumes"
Oct 30 21:22:59.161551 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:22:59.161515    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:23:00.057082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057052    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"]
Oct 30 21:23:00.057676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057662    4918 cpu_manager.go:414] "containerMap: removing container" podUID="73e680b3-0724-436c-a33f-217385db1bd1" containerName="payload-pre-processing"
Oct 30 21:23:00.057726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057677    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e680b3-0724-436c-a33f-217385db1bd1" containerName="payload-pre-processing"
Oct 30 21:23:00.057726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057688    4918 cpu_manager.go:414] "containerMap: removing container" podUID="75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167" containerName="revoke-keys"
Oct 30 21:23:00.057726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057693    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167" containerName="revoke-keys"
Oct 30 21:23:00.057726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057704    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" containerName="istio-proxy"
Oct 30 21:23:00.057726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057708    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" containerName="istio-proxy"
Oct 30 21:23:00.057884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057730    4918 cpu_manager.go:414] "containerMap: removing container" podUID="27df2c10-4316-489e-9a7a-19417b131519" containerName="payload-pre-processing"
Oct 30 21:23:00.057884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057734    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27df2c10-4316-489e-9a7a-19417b131519" containerName="payload-pre-processing"
Oct 30 21:23:00.057884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057745    4918 cpu_manager.go:414] "containerMap: removing container" podUID="32638ae5-1e74-47de-b9ed-2c04e3811409" containerName="istio-proxy"
Oct 30 21:23:00.057884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057750    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32638ae5-1e74-47de-b9ed-2c04e3811409" containerName="istio-proxy"
Oct 30 21:23:00.057884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057760    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6012c0fa-471a-41ec-ba1a-95be1768af03" containerName="payload-processing"
Oct 30 21:23:00.057884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057764    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6012c0fa-471a-41ec-ba1a-95be1768af03" containerName="payload-processing"
Oct 30 21:23:00.057884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057790    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e3fe60da-e4fe-46f8-9006-d8097a0a3ef6" containerName="payload-processing"
Oct 30 21:23:00.057884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057794    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3fe60da-e4fe-46f8-9006-d8097a0a3ef6" containerName="payload-processing"
Oct 30 21:23:00.057884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057879    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="27df2c10-4316-489e-9a7a-19417b131519" containerName="payload-pre-processing"
Oct 30 21:23:00.058134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057891    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="32638ae5-1e74-47de-b9ed-2c04e3811409" containerName="istio-proxy"
Oct 30 21:23:00.058134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057898    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6012c0fa-471a-41ec-ba1a-95be1768af03" containerName="payload-processing"
Oct 30 21:23:00.058134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057906    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3fe60da-e4fe-46f8-9006-d8097a0a3ef6" containerName="payload-processing"
Oct 30 21:23:00.058134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057916    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e680b3-0724-436c-a33f-217385db1bd1" containerName="payload-pre-processing"
Oct 30 21:23:00.058134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057924    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167" containerName="revoke-keys"
Oct 30 21:23:00.058134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.057930    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40df71b-5ccf-4655-bee6-eec8cf48d47b" containerName="istio-proxy"
Oct 30 21:23:00.062921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.062904    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:00.067712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.067686    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:00.067848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.067779    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:00.070425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.070367    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:00.070622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.070610    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:23:00.071766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.071540    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"]
Oct 30 21:23:00.132486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.132445    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwsp7\" (UniqueName: \"kubernetes.io/projected/016219dc-2659-479d-b003-1af3fb5c9e78-kube-api-access-jwsp7\") pod \"maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6\" (UID: \"016219dc-2659-479d-b003-1af3fb5c9e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:00.132659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.132541    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6\" (UID: \"016219dc-2659-479d-b003-1af3fb5c9e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:00.132798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.132779    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6\" (UID: \"016219dc-2659-479d-b003-1af3fb5c9e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:00.229254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.229224    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"]
Oct 30 21:23:00.233832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.233809    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.234851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.234153    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwsp7\" (UniqueName: \"kubernetes.io/projected/016219dc-2659-479d-b003-1af3fb5c9e78-kube-api-access-jwsp7\") pod \"maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6\" (UID: \"016219dc-2659-479d-b003-1af3fb5c9e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:00.234988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.234967    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6\" (UID: \"016219dc-2659-479d-b003-1af3fb5c9e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:00.235071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.235055    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6\" (UID: \"016219dc-2659-479d-b003-1af3fb5c9e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:00.235870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.235850    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6\" (UID: \"016219dc-2659-479d-b003-1af3fb5c9e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:00.238538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.238517    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6\" (UID: \"016219dc-2659-479d-b003-1af3fb5c9e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:00.242709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.242685    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"]
Oct 30 21:23:00.243551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.243531    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-6563b6\"/\"kube-root-ca.crt\""
Oct 30 21:23:00.243551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.243531    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-6563b6\"/\"openshift-service-ca.crt\""
Oct 30 21:23:00.247409 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.247372    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwsp7\" (UniqueName: \"kubernetes.io/projected/016219dc-2659-479d-b003-1af3fb5c9e78-kube-api-access-jwsp7\") pod \"maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6\" (UID: \"016219dc-2659-479d-b003-1af3fb5c9e78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:00.270407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.270363    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-6563b6\"/\"default-dockercfg-9zf67\""
Oct 30 21:23:00.270407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.270375    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-6563b6\"/\"praxis-route-6563b6-kserve-self-signed-certs\""
Oct 30 21:23:00.336182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.336115    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tls-certs\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.336182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.336166    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-dshm\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.336432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.336315    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-home\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.336510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.336440    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-model-cache\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.336510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.336500    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tmp-dir\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.336596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.336538    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7f4\" (UniqueName: \"kubernetes.io/projected/ed90e959-7770-4c1e-82fd-3abfd1800ce2-kube-api-access-rb7f4\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.380733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.380709    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:00.450427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.442691    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-model-cache\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.450427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.442748    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tmp-dir\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.450427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.442791    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7f4\" (UniqueName: \"kubernetes.io/projected/ed90e959-7770-4c1e-82fd-3abfd1800ce2-kube-api-access-rb7f4\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.450427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.442900    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tls-certs\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.450427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.442940    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-dshm\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.450427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.443038    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-home\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.450427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.447034    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-model-cache\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.450427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.447279    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tmp-dir\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.460429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.459509    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-home\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.465430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.462622    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-dshm\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.465430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.463278    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tls-certs\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.488431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.486524    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7f4\" (UniqueName: \"kubernetes.io/projected/ed90e959-7770-4c1e-82fd-3abfd1800ce2-kube-api-access-rb7f4\") pod \"praxis-route-6563b6-kserve-78ff97b85c-qt96d\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") " pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.585980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.585921    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:00.876781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:00.876761    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"]
Oct 30 21:23:01.092620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:01.092597    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"]
Oct 30 21:23:01.099694 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:23:01.099668    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded90e959_7770_4c1e_82fd_3abfd1800ce2.slice/crio-c566715b77bac41674825de176f935e68d5604d2ef5a18bfb87f725556ae2a1c WatchSource:0}: Error finding container c566715b77bac41674825de176f935e68d5604d2ef5a18bfb87f725556ae2a1c: Status 404 returned error can't find the container with id c566715b77bac41674825de176f935e68d5604d2ef5a18bfb87f725556ae2a1c
Oct 30 21:23:01.782471 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:01.782432    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f\": RecentStats: unable to find data in memory cache]"
Oct 30 21:23:01.873414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:01.870100    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6" event={"ID":"016219dc-2659-479d-b003-1af3fb5c9e78","Type":"ContainerStarted","Data":"708d4f973f9eff87ba2ac195bf894e09c889abf49e9d2e0cfd3be20633594e87"}
Oct 30 21:23:01.873414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:01.870166    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6" event={"ID":"016219dc-2659-479d-b003-1af3fb5c9e78","Type":"ContainerStarted","Data":"d2ac6cd12162e491c8a3d20d142fc00398c1ecd21f6b8d44810693bd2b0643cd"}
Oct 30 21:23:01.873414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:01.872194    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d" event={"ID":"ed90e959-7770-4c1e-82fd-3abfd1800ce2","Type":"ContainerStarted","Data":"3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c"}
Oct 30 21:23:01.873414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:01.872223    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d" event={"ID":"ed90e959-7770-4c1e-82fd-3abfd1800ce2","Type":"ContainerStarted","Data":"c566715b77bac41674825de176f935e68d5604d2ef5a18bfb87f725556ae2a1c"}
Oct 30 21:23:01.914414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:01.911465    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6" podStartSLOduration=1.911439812 podStartE2EDuration="1.911439812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:01.892195003 +0000 UTC m=+1294.938754148" watchObservedRunningTime="2026-10-30 21:23:01.911439812 +0000 UTC m=+1294.957998963"
Oct 30 21:23:01.950418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:01.917195    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d" podStartSLOduration=1.917172691 podStartE2EDuration="1.917172691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:01.914999201 +0000 UTC m=+1294.961558345" watchObservedRunningTime="2026-10-30 21:23:01.917172691 +0000 UTC m=+1294.963731837"
Oct 30 21:23:02.599414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:02.589487    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:02.922780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:02.922665    4918 generic.go:356] "Generic (PLEG): container finished" podID="016219dc-2659-479d-b003-1af3fb5c9e78" containerID="708d4f973f9eff87ba2ac195bf894e09c889abf49e9d2e0cfd3be20633594e87" exitCode=0
Oct 30 21:23:02.922956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:02.922808    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6" event={"ID":"016219dc-2659-479d-b003-1af3fb5c9e78","Type":"ContainerDied","Data":"708d4f973f9eff87ba2ac195bf894e09c889abf49e9d2e0cfd3be20633594e87"}
Oct 30 21:23:02.984140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:02.984103    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"]
Oct 30 21:23:02.984423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:02.984380    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k" podUID="536e37a1-058c-42e5-8b80-7cc2d4194711" containerName="payload-pre-processing" containerID="cri-o://96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa" gracePeriod=30
Oct 30 21:23:03.543625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.543596    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"]
Oct 30 21:23:03.549211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.549186    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09eb50-tn9gd"]
Oct 30 21:23:03.639576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.639285    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"
Oct 30 21:23:03.775415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.753772    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536e37a1-058c-42e5-8b80-7cc2d4194711-kube-api-access-jrxlr\" (UniqueName: \"kubernetes.io/projected/536e37a1-058c-42e5-8b80-7cc2d4194711-kube-api-access-jrxlr\") pod \"536e37a1-058c-42e5-8b80-7cc2d4194711\" (UID: \"536e37a1-058c-42e5-8b80-7cc2d4194711\") "
Oct 30 21:23:03.791534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.776010    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536e37a1-058c-42e5-8b80-7cc2d4194711-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536e37a1-058c-42e5-8b80-7cc2d4194711-plugins-config-volume\") pod \"536e37a1-058c-42e5-8b80-7cc2d4194711\" (UID: \"536e37a1-058c-42e5-8b80-7cc2d4194711\") "
Oct 30 21:23:03.793278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.793245    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536e37a1-058c-42e5-8b80-7cc2d4194711-plugins-config-volume" pod "536e37a1-058c-42e5-8b80-7cc2d4194711" (UID: "536e37a1-058c-42e5-8b80-7cc2d4194711"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:03.835667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.835610    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536e37a1-058c-42e5-8b80-7cc2d4194711-kube-api-access-jrxlr" pod "536e37a1-058c-42e5-8b80-7cc2d4194711" (UID: "536e37a1-058c-42e5-8b80-7cc2d4194711"). InnerVolumeSpecName "kube-api-access-jrxlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:03.911457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.904565    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536e37a1-058c-42e5-8b80-7cc2d4194711-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:03.911457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.904610    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrxlr\" (UniqueName: \"kubernetes.io/projected/536e37a1-058c-42e5-8b80-7cc2d4194711-kube-api-access-jrxlr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:03.932179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.932148    4918 generic.go:356] "Generic (PLEG): container finished" podID="536e37a1-058c-42e5-8b80-7cc2d4194711" containerID="96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa" exitCode=0
Oct 30 21:23:03.932368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.932198    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"
Oct 30 21:23:03.932368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.932238    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k" event={"ID":"536e37a1-058c-42e5-8b80-7cc2d4194711","Type":"ContainerDied","Data":"96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa"}
Oct 30 21:23:03.932368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.932286    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k" event={"ID":"536e37a1-058c-42e5-8b80-7cc2d4194711","Type":"ContainerDied","Data":"ef7fd49c9bac1390ea9dbea722ad317cd226297c67de6a1bef0b426547c57260"}
Oct 30 21:23:03.932368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.932314    4918 scope.go:122] "RemoveContainer" containerID="96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa"
Oct 30 21:23:03.945629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.945607    4918 scope.go:122] "RemoveContainer" containerID="96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa"
Oct 30 21:23:03.945922 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:03.945899    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa\": container with ID starting with 96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa not found: ID does not exist" containerID="96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa"
Oct 30 21:23:03.946010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.945929    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa"} err="failed to get container status \"96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa\": rpc error: code = NotFound desc = could not find container \"96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa\": container with ID starting with 96b051538e780e4e807fad11214ab9763da243b5e352ac5917f00e09ca29e0aa not found: ID does not exist"
Oct 30 21:23:03.964729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.964692    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"]
Oct 30 21:23:03.970777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:03.970745    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09eb50-69f6b49db7-2zv9k"]
Oct 30 21:23:04.164101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.164077    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:04.322571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.322543    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016219dc-2659-479d-b003-1af3fb5c9e78-kube-api-access-jwsp7\" (UniqueName: \"kubernetes.io/projected/016219dc-2659-479d-b003-1af3fb5c9e78-kube-api-access-jwsp7\") pod \"016219dc-2659-479d-b003-1af3fb5c9e78\" (UID: \"016219dc-2659-479d-b003-1af3fb5c9e78\") "
Oct 30 21:23:04.322763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.322751    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-openshift-ca\") pod \"016219dc-2659-479d-b003-1af3fb5c9e78\" (UID: \"016219dc-2659-479d-b003-1af3fb5c9e78\") "
Oct 30 21:23:04.322839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.322827    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-xks-ca\") pod \"016219dc-2659-479d-b003-1af3fb5c9e78\" (UID: \"016219dc-2659-479d-b003-1af3fb5c9e78\") "
Oct 30 21:23:04.323126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.323098    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-openshift-ca" pod "016219dc-2659-479d-b003-1af3fb5c9e78" (UID: "016219dc-2659-479d-b003-1af3fb5c9e78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:04.323315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.323281    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:04.324847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.324823    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016219dc-2659-479d-b003-1af3fb5c9e78-kube-api-access-jwsp7" pod "016219dc-2659-479d-b003-1af3fb5c9e78" (UID: "016219dc-2659-479d-b003-1af3fb5c9e78"). InnerVolumeSpecName "kube-api-access-jwsp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:04.325025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.325004    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-xks-ca" pod "016219dc-2659-479d-b003-1af3fb5c9e78" (UID: "016219dc-2659-479d-b003-1af3fb5c9e78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:04.424630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.424554    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwsp7\" (UniqueName: \"kubernetes.io/projected/016219dc-2659-479d-b003-1af3fb5c9e78-kube-api-access-jwsp7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:04.424630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.424582    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016219dc-2659-479d-b003-1af3fb5c9e78-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:04.944358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.944330    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"
Oct 30 21:23:04.944765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.944367    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6" event={"ID":"016219dc-2659-479d-b003-1af3fb5c9e78","Type":"ContainerDied","Data":"d2ac6cd12162e491c8a3d20d142fc00398c1ecd21f6b8d44810693bd2b0643cd"}
Oct 30 21:23:04.944765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:04.944447    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2ac6cd12162e491c8a3d20d142fc00398c1ecd21f6b8d44810693bd2b0643cd"
Oct 30 21:23:05.077252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:05.077222    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536e37a1-058c-42e5-8b80-7cc2d4194711" path="/var/lib/kubelet/pods/536e37a1-058c-42e5-8b80-7cc2d4194711/volumes"
Oct 30 21:23:05.077701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:05.077687    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167" path="/var/lib/kubelet/pods/75c8b17c-8a8b-43e0-abbd-5eb5a8b6b167/volumes"
Oct 30 21:23:08.023847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:08.023815    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"]
Oct 30 21:23:08.027225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:08.027196    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d789f639-j96ll"]
Oct 30 21:23:09.083000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:09.082965    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b0d694-f94f-4036-afce-87a92ef93447" path="/var/lib/kubelet/pods/b5b0d694-f94f-4036-afce-87a92ef93447/volumes"
Oct 30 21:23:09.130195 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:09.130159    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:23:09.257865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:09.257831    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"]
Oct 30 21:23:09.261813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:09.261776    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d87b5f-vc2nm"]
Oct 30 21:23:10.191360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:10.191325    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"]
Oct 30 21:23:10.205732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:10.205700    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"]
Oct 30 21:23:10.595125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:10.595096    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:10.719179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:10.718909    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:10.829090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:10.829059    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"]
Oct 30 21:23:10.834323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:10.834295    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-df078c-9mwc6"]
Oct 30 21:23:11.131453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:11.116700    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="016219dc-2659-479d-b003-1af3fb5c9e78" path="/var/lib/kubelet/pods/016219dc-2659-479d-b003-1af3fb5c9e78/volumes"
Oct 30 21:23:11.131453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:11.117381    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7da1b1bd-9254-4036-b851-d814bbfbaaf3" path="/var/lib/kubelet/pods/7da1b1bd-9254-4036-b851-d814bbfbaaf3/volumes"
Oct 30 21:23:11.157870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:11.157377    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:23:13.089633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:13.089595    4918 generic.go:356] "Generic (PLEG): container finished" podID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerID="66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c" exitCode=0
Oct 30 21:23:13.090123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:13.089676    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" event={"ID":"020138d9-94c2-4a2f-98a6-f92f25f661ee","Type":"ContainerDied","Data":"66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c"}
Oct 30 21:23:13.090123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:13.090095    4918 scope.go:122] "RemoveContainer" containerID="66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c"
Oct 30 21:23:13.092224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:13.092187    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:23:13.092570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:13.092545    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:23:13.092775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:13.092756    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:23:14.084684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.084655    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"]
Oct 30 21:23:14.095420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.085718    4918 cpu_manager.go:414] "containerMap: removing container" podUID="016219dc-2659-479d-b003-1af3fb5c9e78" containerName="revoke-keys"
Oct 30 21:23:14.095987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.095963    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016219dc-2659-479d-b003-1af3fb5c9e78" containerName="revoke-keys"
Oct 30 21:23:14.096192 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.096173    4918 cpu_manager.go:414] "containerMap: removing container" podUID="536e37a1-058c-42e5-8b80-7cc2d4194711" containerName="payload-pre-processing"
Oct 30 21:23:14.110671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.110632    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536e37a1-058c-42e5-8b80-7cc2d4194711" containerName="payload-pre-processing"
Oct 30 21:23:14.111148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.111121    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="536e37a1-058c-42e5-8b80-7cc2d4194711" containerName="payload-pre-processing"
Oct 30 21:23:14.111298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.111287    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="016219dc-2659-479d-b003-1af3fb5c9e78" containerName="revoke-keys"
Oct 30 21:23:14.126945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.126651    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.129742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.129714    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"]
Oct 30 21:23:14.132800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.132771    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" event={"ID":"020138d9-94c2-4a2f-98a6-f92f25f661ee","Type":"ContainerStarted","Data":"139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f"}
Oct 30 21:23:14.137973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.137953    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-83921a-openshift-default-dockercfg-7dds9\""
Oct 30 21:23:14.249985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.249956    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-istio-token\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.250156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.250003    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-credential-socket\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.250156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.250116    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-certs\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.250245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.250232    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c07b1b-9c38-4b55-b203-6c922431639c-istio-podinfo\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.250302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.250291    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c07b1b-9c38-4b55-b203-6c922431639c-istiod-ca-cert\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.250353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.250343    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-envoy\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.250465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.250445    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-data\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.250517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.250501    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-socket\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.250673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.250654    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sck88\" (UniqueName: \"kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-kube-api-access-sck88\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.293477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.292547    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:23:14.352934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.352529    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sck88\" (UniqueName: \"kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-kube-api-access-sck88\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.353316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.353291    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-istio-token\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.353758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.353739    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-credential-socket\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.354339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.354322    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-certs\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.359796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.359777    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c07b1b-9c38-4b55-b203-6c922431639c-istio-podinfo\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.359953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.359936    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c07b1b-9c38-4b55-b203-6c922431639c-istiod-ca-cert\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.360073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.360059    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-envoy\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.360225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.360211    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-data\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.367486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.360343    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-socket\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.368023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.359694    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-certs\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.368173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.354237    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-credential-socket\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.395415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.394650    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-socket\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.395415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.394924    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-data\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.399501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.399470    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c07b1b-9c38-4b55-b203-6c922431639c-istiod-ca-cert\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.413412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.402732    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-envoy\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.413412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.409860    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-istio-token\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.413412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.411457    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sck88\" (UniqueName: \"kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-kube-api-access-sck88\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.445004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.444972    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c07b1b-9c38-4b55-b203-6c922431639c-istio-podinfo\") pod \"e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") " pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:14.518302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.518129    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:23:14.523056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.523013    4918 patch_prober.go:29] interesting pod/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:23:14.523230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.523082    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 21:23:14.744945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:14.744913    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:15.130285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.117374    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"]
Oct 30 21:23:15.131734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.131702    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:23:15.131892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.131878    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:23:15.131990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.131979    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:23:15.145692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.145668    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" event={"ID":"83c07b1b-9c38-4b55-b203-6c922431639c","Type":"ContainerStarted","Data":"8811f2d6f1d42c0204a482e5f9dffe6959486a1f14ec6946e8bad29ee29a3df0"}
Oct 30 21:23:15.210411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.209494    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn" podUID="27047a93-6fc0-4006-80a4-dbef6512129b" containerName="payload-processing" containerID="cri-o://7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394" gracePeriod=25
Oct 30 21:23:15.212679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.212453    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w" podUID="b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae" containerName="payload-pre-processing" containerID="cri-o://d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b" gracePeriod=25
Oct 30 21:23:15.517418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.516800    4918 patch_prober.go:29] interesting pod/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:23:15.517418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.516861    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 21:23:15.638616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.638552    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"
Oct 30 21:23:15.720419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.716092    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-kube-api-access-xzhxr\" (UniqueName: \"kubernetes.io/projected/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-kube-api-access-xzhxr\") pod \"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae\" (UID: \"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae\") "
Oct 30 21:23:15.720419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.716208    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-plugins-config-volume\") pod \"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae\" (UID: \"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae\") "
Oct 30 21:23:15.747415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.738701    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-plugins-config-volume" pod "b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae" (UID: "b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:15.808099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.808046    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-kube-api-access-xzhxr" pod "b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae" (UID: "b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae"). InnerVolumeSpecName "kube-api-access-xzhxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:15.828430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.820575    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"
Oct 30 21:23:15.833446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.830804    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzhxr\" (UniqueName: \"kubernetes.io/projected/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-kube-api-access-xzhxr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:15.833698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.833664    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:15.935270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.935188    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27047a93-6fc0-4006-80a4-dbef6512129b-kube-api-access-g72fs\" (UniqueName: \"kubernetes.io/projected/27047a93-6fc0-4006-80a4-dbef6512129b-kube-api-access-g72fs\") pod \"27047a93-6fc0-4006-80a4-dbef6512129b\" (UID: \"27047a93-6fc0-4006-80a4-dbef6512129b\") "
Oct 30 21:23:15.935270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.935249    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27047a93-6fc0-4006-80a4-dbef6512129b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27047a93-6fc0-4006-80a4-dbef6512129b-plugins-config-volume\") pod \"27047a93-6fc0-4006-80a4-dbef6512129b\" (UID: \"27047a93-6fc0-4006-80a4-dbef6512129b\") "
Oct 30 21:23:15.935702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.935679    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27047a93-6fc0-4006-80a4-dbef6512129b-plugins-config-volume" pod "27047a93-6fc0-4006-80a4-dbef6512129b" (UID: "27047a93-6fc0-4006-80a4-dbef6512129b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:15.937561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:15.937538    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27047a93-6fc0-4006-80a4-dbef6512129b-kube-api-access-g72fs" pod "27047a93-6fc0-4006-80a4-dbef6512129b" (UID: "27047a93-6fc0-4006-80a4-dbef6512129b"). InnerVolumeSpecName "kube-api-access-g72fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:16.036602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.036567    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g72fs\" (UniqueName: \"kubernetes.io/projected/27047a93-6fc0-4006-80a4-dbef6512129b-kube-api-access-g72fs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:16.036602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.036596    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27047a93-6fc0-4006-80a4-dbef6512129b-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:16.155105 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.155074    4918 generic.go:356] "Generic (PLEG): container finished" podID="27047a93-6fc0-4006-80a4-dbef6512129b" containerID="7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394" exitCode=0
Oct 30 21:23:16.155620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.155124    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"
Oct 30 21:23:16.155620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.155161    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn" event={"ID":"27047a93-6fc0-4006-80a4-dbef6512129b","Type":"ContainerDied","Data":"7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394"}
Oct 30 21:23:16.155620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.155220    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn" event={"ID":"27047a93-6fc0-4006-80a4-dbef6512129b","Type":"ContainerDied","Data":"be3320fa866602218483ebda79ffece540165b3248cbf83ce590791c9d4ea01e"}
Oct 30 21:23:16.155620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.155252    4918 scope.go:122] "RemoveContainer" containerID="7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394"
Oct 30 21:23:16.156877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.156855    4918 generic.go:356] "Generic (PLEG): container finished" podID="b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae" containerID="d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b" exitCode=0
Oct 30 21:23:16.156952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.156895    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"
Oct 30 21:23:16.157098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.156954    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w" event={"ID":"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae","Type":"ContainerDied","Data":"d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b"}
Oct 30 21:23:16.157186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.157171    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w" event={"ID":"b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae","Type":"ContainerDied","Data":"d9dcdfe5964e5cc04ea05d9399f8ed167eb12406229e7eb760997763b334d696"}
Oct 30 21:23:16.159286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.159260    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" event={"ID":"83c07b1b-9c38-4b55-b203-6c922431639c","Type":"ContainerStarted","Data":"67cb28f4dd506e044b9e9573b13dec93952f377ad4e90878b7d283735915c0b6"}
Oct 30 21:23:16.168416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.168380    4918 scope.go:122] "RemoveContainer" containerID="7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394"
Oct 30 21:23:16.169101 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:16.169074    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394\": container with ID starting with 7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394 not found: ID does not exist" containerID="7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394"
Oct 30 21:23:16.169260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.169229    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394"} err="failed to get container status \"7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394\": rpc error: code = NotFound desc = could not find container \"7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394\": container with ID starting with 7528018544c9969c88fb914ca11a302f9350f2ef3f31388ad4504c2cf8256394 not found: ID does not exist"
Oct 30 21:23:16.169380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.169262    4918 scope.go:122] "RemoveContainer" containerID="d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b"
Oct 30 21:23:16.198007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.185486    4918 scope.go:122] "RemoveContainer" containerID="d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b"
Oct 30 21:23:16.221436 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:16.216654    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b\": container with ID starting with d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b not found: ID does not exist" containerID="d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b"
Oct 30 21:23:16.221436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.216700    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b"} err="failed to get container status \"d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b\": rpc error: code = NotFound desc = could not find container \"d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b\": container with ID starting with d6488a4fa45a637c2599b49c5d1aa3e896a0f712b6f0a896a263fc1539048c0b not found: ID does not exist"
Oct 30 21:23:16.221436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.220471    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" podStartSLOduration=2.220455617 podStartE2EDuration="2.220455617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:16.216831195 +0000 UTC m=+1309.263390368" watchObservedRunningTime="2026-10-30 21:23:16.220455617 +0000 UTC m=+1309.267014834"
Oct 30 21:23:16.235189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.235167    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"]
Oct 30 21:23:16.238618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.238596    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-df078c-7456696599-6fj4w"]
Oct 30 21:23:16.249966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.249943    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"]
Oct 30 21:23:16.256199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.256179    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-df078c-85677dbbdc-s9vkn"]
Oct 30 21:23:16.516621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.516583    4918 patch_prober.go:29] interesting pod/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:23:16.517663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.517585    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 21:23:16.745585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.745548    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:16.751167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:16.751134    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:16.873135 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:16.873061    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f\": RecentStats: unable to find data in memory cache]"
Oct 30 21:23:17.112454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:17.107712    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27047a93-6fc0-4006-80a4-dbef6512129b" path="/var/lib/kubelet/pods/27047a93-6fc0-4006-80a4-dbef6512129b/volumes"
Oct 30 21:23:17.113574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:17.113546    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae" path="/var/lib/kubelet/pods/b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae/volumes"
Oct 30 21:23:17.189205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:17.189113    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:17.206988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:17.206669    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:23:17.641424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:17.638271    4918 patch_prober.go:29] interesting pod/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:23:17.641424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:17.638344    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 21:23:18.516162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:18.516128    4918 patch_prober.go:29] interesting pod/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:23:18.516658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:18.516217    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 21:23:19.156522 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:19.156487    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:23:19.515412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:19.515362    4918 patch_prober.go:29] interesting pod/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 30 21:23:19.515624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:19.515442    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 30 21:23:20.275213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.275181    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"]
Oct 30 21:23:20.275776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.275761    4918 cpu_manager.go:414] "containerMap: removing container" podUID="27047a93-6fc0-4006-80a4-dbef6512129b" containerName="payload-processing"
Oct 30 21:23:20.275827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.275777    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27047a93-6fc0-4006-80a4-dbef6512129b" containerName="payload-processing"
Oct 30 21:23:20.275827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.275808    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae" containerName="payload-pre-processing"
Oct 30 21:23:20.275827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.275815    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae" containerName="payload-pre-processing"
Oct 30 21:23:20.275938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.275908    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b437c9-b4ab-43e5-a0a6-ca794dbc71ae" containerName="payload-pre-processing"
Oct 30 21:23:20.275938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.275920    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="27047a93-6fc0-4006-80a4-dbef6512129b" containerName="payload-processing"
Oct 30 21:23:20.281164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.281144    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"
Oct 30 21:23:20.291718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.291690    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-83921a\""
Oct 30 21:23:20.293013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.292981    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-83921a-dockercfg-x5cxf\""
Oct 30 21:23:20.295469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.295443    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"]
Oct 30 21:23:20.299763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.299740    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"]
Oct 30 21:23:20.305276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.305253    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:20.315317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.315297    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"]
Oct 30 21:23:20.345066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.345040    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"]
Oct 30 21:23:20.349191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.349175    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"
Oct 30 21:23:20.356348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.356322    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"]
Oct 30 21:23:20.398126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.391740    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:20.422416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.402645    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:20.422416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.402794    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:20.422416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.403130    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:23:20.422416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.411102    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e21012-166b-4a6d-a762-1c1f2ad9d66b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk\" (UID: \"98e21012-166b-4a6d-a762-1c1f2ad9d66b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"
Oct 30 21:23:20.422416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.411149    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq\" (UID: \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:20.422416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.411260    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1aaa42-4b18-4050-84f7-baaa4947b1af-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq\" (UID: \"ff1aaa42-4b18-4050-84f7-baaa4947b1af\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"
Oct 30 21:23:20.422416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.411295    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq\" (UID: \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:20.422416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.411341    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcfh\" (UniqueName: \"kubernetes.io/projected/72c3da8c-1fb8-47b9-8daf-3427b62e1918-kube-api-access-kvcfh\") pod \"maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq\" (UID: \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:20.422416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.411384    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfw7t\" (UniqueName: \"kubernetes.io/projected/98e21012-166b-4a6d-a762-1c1f2ad9d66b-kube-api-access-vfw7t\") pod \"payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk\" (UID: \"98e21012-166b-4a6d-a762-1c1f2ad9d66b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"
Oct 30 21:23:20.422416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.411449    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cm7g\" (UniqueName: \"kubernetes.io/projected/ff1aaa42-4b18-4050-84f7-baaa4947b1af-kube-api-access-7cm7g\") pod \"payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq\" (UID: \"ff1aaa42-4b18-4050-84f7-baaa4947b1af\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"
Oct 30 21:23:20.518418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.514248    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1aaa42-4b18-4050-84f7-baaa4947b1af-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq\" (UID: \"ff1aaa42-4b18-4050-84f7-baaa4947b1af\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"
Oct 30 21:23:20.518418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.514303    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq\" (UID: \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:20.518418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.514362    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcfh\" (UniqueName: \"kubernetes.io/projected/72c3da8c-1fb8-47b9-8daf-3427b62e1918-kube-api-access-kvcfh\") pod \"maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq\" (UID: \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:20.525457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.519324    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfw7t\" (UniqueName: \"kubernetes.io/projected/98e21012-166b-4a6d-a762-1c1f2ad9d66b-kube-api-access-vfw7t\") pod \"payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk\" (UID: \"98e21012-166b-4a6d-a762-1c1f2ad9d66b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"
Oct 30 21:23:20.525457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.519431    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cm7g\" (UniqueName: \"kubernetes.io/projected/ff1aaa42-4b18-4050-84f7-baaa4947b1af-kube-api-access-7cm7g\") pod \"payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq\" (UID: \"ff1aaa42-4b18-4050-84f7-baaa4947b1af\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"
Oct 30 21:23:20.525457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.519607    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e21012-166b-4a6d-a762-1c1f2ad9d66b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk\" (UID: \"98e21012-166b-4a6d-a762-1c1f2ad9d66b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"
Oct 30 21:23:20.525457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.519641    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq\" (UID: \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:20.525457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.522535    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq\" (UID: \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:20.546751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.543600    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1aaa42-4b18-4050-84f7-baaa4947b1af-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq\" (UID: \"ff1aaa42-4b18-4050-84f7-baaa4947b1af\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"
Oct 30 21:23:20.586950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.586922    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:23:20.590773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.590749    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:23:20.594036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.594012    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq\" (UID: \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:20.594475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.594450    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cm7g\" (UniqueName: \"kubernetes.io/projected/ff1aaa42-4b18-4050-84f7-baaa4947b1af-kube-api-access-7cm7g\") pod \"payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq\" (UID: \"ff1aaa42-4b18-4050-84f7-baaa4947b1af\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"
Oct 30 21:23:20.594940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.594918    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e21012-166b-4a6d-a762-1c1f2ad9d66b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk\" (UID: \"98e21012-166b-4a6d-a762-1c1f2ad9d66b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"
Oct 30 21:23:20.601016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.596914    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"
Oct 30 21:23:20.619970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.616278    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfw7t\" (UniqueName: \"kubernetes.io/projected/98e21012-166b-4a6d-a762-1c1f2ad9d66b-kube-api-access-vfw7t\") pod \"payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk\" (UID: \"98e21012-166b-4a6d-a762-1c1f2ad9d66b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"
Oct 30 21:23:20.619970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.618944    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcfh\" (UniqueName: \"kubernetes.io/projected/72c3da8c-1fb8-47b9-8daf-3427b62e1918-kube-api-access-kvcfh\") pod \"maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq\" (UID: \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:20.627822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.627800    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:20.665947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.665750    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"
Oct 30 21:23:20.941403 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.941354    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"]
Oct 30 21:23:20.949555 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:23:20.949529    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1aaa42_4b18_4050_84f7_baaa4947b1af.slice/crio-dbec71f541237af560ea1aa7dca917c444483d145337905f6f1d48fdc89f15fb WatchSource:0}: Error finding container dbec71f541237af560ea1aa7dca917c444483d145337905f6f1d48fdc89f15fb: Status 404 returned error can't find the container with id dbec71f541237af560ea1aa7dca917c444483d145337905f6f1d48fdc89f15fb
Oct 30 21:23:20.972907 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:23:20.972855    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c3da8c_1fb8_47b9_8daf_3427b62e1918.slice/crio-00c2abc576f69ca3407d0581b8fc2cd0e83f12a0cd04cdd4492690837955bddc WatchSource:0}: Error finding container 00c2abc576f69ca3407d0581b8fc2cd0e83f12a0cd04cdd4492690837955bddc: Status 404 returned error can't find the container with id 00c2abc576f69ca3407d0581b8fc2cd0e83f12a0cd04cdd4492690837955bddc
Oct 30 21:23:20.974268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:20.974243    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"]
Oct 30 21:23:21.004308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:21.004279    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"]
Oct 30 21:23:21.006663 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:23:21.006627    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e21012_166b_4a6d_a762_1c1f2ad9d66b.slice/crio-6ea23508ee57969c095e4c16c9afc7ceac188f0eb1a3a9cee3597efa65cad401 WatchSource:0}: Error finding container 6ea23508ee57969c095e4c16c9afc7ceac188f0eb1a3a9cee3597efa65cad401: Status 404 returned error can't find the container with id 6ea23508ee57969c095e4c16c9afc7ceac188f0eb1a3a9cee3597efa65cad401
Oct 30 21:23:21.254699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:21.254664    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq" event={"ID":"72c3da8c-1fb8-47b9-8daf-3427b62e1918","Type":"ContainerStarted","Data":"00c2abc576f69ca3407d0581b8fc2cd0e83f12a0cd04cdd4492690837955bddc"}
Oct 30 21:23:21.256454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:21.256426    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk" event={"ID":"98e21012-166b-4a6d-a762-1c1f2ad9d66b","Type":"ContainerStarted","Data":"b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda"}
Oct 30 21:23:21.256605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:21.256472    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk" event={"ID":"98e21012-166b-4a6d-a762-1c1f2ad9d66b","Type":"ContainerStarted","Data":"6ea23508ee57969c095e4c16c9afc7ceac188f0eb1a3a9cee3597efa65cad401"}
Oct 30 21:23:21.256605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:21.256517    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"
Oct 30 21:23:21.258119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:21.258097    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq" event={"ID":"ff1aaa42-4b18-4050-84f7-baaa4947b1af","Type":"ContainerStarted","Data":"0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f"}
Oct 30 21:23:21.258201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:21.258140    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq" event={"ID":"ff1aaa42-4b18-4050-84f7-baaa4947b1af","Type":"ContainerStarted","Data":"dbec71f541237af560ea1aa7dca917c444483d145337905f6f1d48fdc89f15fb"}
Oct 30 21:23:21.258306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:21.258291    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"
Oct 30 21:23:21.301334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:21.301290    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk" podStartSLOduration=1.301275901 podStartE2EDuration="1.301275901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:21.300563005 +0000 UTC m=+1314.347122169" watchObservedRunningTime="2026-10-30 21:23:21.301275901 +0000 UTC m=+1314.347835111"
Oct 30 21:23:21.318819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:21.318760    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq" podStartSLOduration=1.318746335 podStartE2EDuration="1.318746335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:21.318183006 +0000 UTC m=+1314.364742177" watchObservedRunningTime="2026-10-30 21:23:21.318746335 +0000 UTC m=+1314.365305491"
Oct 30 21:23:22.267752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:22.267658    4918 generic.go:356] "Generic (PLEG): container finished" podID="72c3da8c-1fb8-47b9-8daf-3427b62e1918" containerID="85373d4736dddc57b88924bb306e6646f0f322a47dd0b4c54f8063a36f00a2f7" exitCode=0
Oct 30 21:23:22.267927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:22.267746    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq" event={"ID":"72c3da8c-1fb8-47b9-8daf-3427b62e1918","Type":"ContainerDied","Data":"85373d4736dddc57b88924bb306e6646f0f322a47dd0b4c54f8063a36f00a2f7"}
Oct 30 21:23:23.441777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:23.441751    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:23.482711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:23.482678    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-xks-ca\") pod \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\" (UID: \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\") "
Oct 30 21:23:23.482879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:23.482858    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c3da8c-1fb8-47b9-8daf-3427b62e1918-kube-api-access-kvcfh\" (UniqueName: \"kubernetes.io/projected/72c3da8c-1fb8-47b9-8daf-3427b62e1918-kube-api-access-kvcfh\") pod \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\" (UID: \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\") "
Oct 30 21:23:23.482953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:23.482928    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-openshift-ca\") pod \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\" (UID: \"72c3da8c-1fb8-47b9-8daf-3427b62e1918\") "
Oct 30 21:23:23.483442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:23.483373    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-openshift-ca" pod "72c3da8c-1fb8-47b9-8daf-3427b62e1918" (UID: "72c3da8c-1fb8-47b9-8daf-3427b62e1918"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:23.483689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:23.483667    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:23.485059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:23.485029    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-xks-ca" pod "72c3da8c-1fb8-47b9-8daf-3427b62e1918" (UID: "72c3da8c-1fb8-47b9-8daf-3427b62e1918"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:23.485617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:23.485596    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c3da8c-1fb8-47b9-8daf-3427b62e1918-kube-api-access-kvcfh" pod "72c3da8c-1fb8-47b9-8daf-3427b62e1918" (UID: "72c3da8c-1fb8-47b9-8daf-3427b62e1918"). InnerVolumeSpecName "kube-api-access-kvcfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:23.584959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:23.584887    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvcfh\" (UniqueName: \"kubernetes.io/projected/72c3da8c-1fb8-47b9-8daf-3427b62e1918-kube-api-access-kvcfh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:23.584959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:23.584915    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72c3da8c-1fb8-47b9-8daf-3427b62e1918-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:24.287178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:24.287137    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq" event={"ID":"72c3da8c-1fb8-47b9-8daf-3427b62e1918","Type":"ContainerDied","Data":"00c2abc576f69ca3407d0581b8fc2cd0e83f12a0cd04cdd4492690837955bddc"}
Oct 30 21:23:24.287178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:24.287168    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"
Oct 30 21:23:24.287381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:24.287205    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00c2abc576f69ca3407d0581b8fc2cd0e83f12a0cd04cdd4492690837955bddc"
Oct 30 21:23:24.954836 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:24.954797    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f\": RecentStats: unable to find data in memory cache]"
Oct 30 21:23:24.955202 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:24.954865    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice/crio-2ab3d842082b7b0786101702fd7d7a5c1dd740c265ffb15acfeefd962eab8b1f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae65e4e9_386b_440c_b568_2402f61346b4.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:23:29.161016 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:29.160978    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:23:29.521903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:29.521870    4918 scope.go:122] "RemoveContainer" containerID="33debf2c280668a06e042c34ff180048310b854aa53bdee4628bba2ba391aa74"
Oct 30 21:23:29.533109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:29.533092    4918 scope.go:122] "RemoveContainer" containerID="178dde54e9c8c03b7b693fe4924844720c22e0e66ca8c4e3f3dfabd5ce2f4571"
Oct 30 21:23:29.542725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:29.542706    4918 scope.go:122] "RemoveContainer" containerID="8c5cc264f3f94f219169be1bb5546b804acfa72aaaa18905f59fc495f308afc2"
Oct 30 21:23:29.552079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:29.552062    4918 scope.go:122] "RemoveContainer" containerID="6d30f1b545d164381783e94add2e61eb48f27d6d745e1168bc7f521cec6cc466"
Oct 30 21:23:29.560919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:29.560899    4918 scope.go:122] "RemoveContainer" containerID="17dfb9453ae483209de5219919973daf2de7fdea409c70e8f14fde6d0181b602"
Oct 30 21:23:30.021624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:30.021589    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"]
Oct 30 21:23:30.024751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:30.024730    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91cbf037-7dddv"]
Oct 30 21:23:31.078579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:31.078540    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0609cc-463d-401e-9117-754584f74c9a" path="/var/lib/kubelet/pods/fe0609cc-463d-401e-9117-754584f74c9a/volumes"
Oct 30 21:23:32.275219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:32.275192    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"
Oct 30 21:23:32.275736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:32.275483    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"
Oct 30 21:23:34.408299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:34.408265    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"]
Oct 30 21:23:34.412607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:34.412578    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-80e936-xtxlq"]
Oct 30 21:23:35.079934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:35.079902    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c3da8c-1fb8-47b9-8daf-3427b62e1918" path="/var/lib/kubelet/pods/72c3da8c-1fb8-47b9-8daf-3427b62e1918/volumes"
Oct 30 21:23:42.575115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:42.575080    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"]
Oct 30 21:23:42.575659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:42.575361    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy" containerID="cri-o://139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f" gracePeriod=30
Oct 30 21:23:45.847508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.847468    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"]
Oct 30 21:23:45.848322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.848295    4918 cpu_manager.go:414] "containerMap: removing container" podUID="72c3da8c-1fb8-47b9-8daf-3427b62e1918" containerName="revoke-keys"
Oct 30 21:23:45.848322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.848321    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c3da8c-1fb8-47b9-8daf-3427b62e1918" containerName="revoke-keys"
Oct 30 21:23:45.848591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.848577    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c3da8c-1fb8-47b9-8daf-3427b62e1918" containerName="revoke-keys"
Oct 30 21:23:45.852146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.852128    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:45.861880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.861858    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:45.862014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.861946    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:23:45.865301 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.865280    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"]
Oct 30 21:23:45.916001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.915976    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:45.916001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.916005    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:45.931105 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.931080    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b8ef01-pzl5h\" (UID: \"e8352713-f7bb-4055-a810-b969caf48a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:45.931261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.931124    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b8ef01-pzl5h\" (UID: \"e8352713-f7bb-4055-a810-b969caf48a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:45.931356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:45.931335    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2t6\" (UniqueName: \"kubernetes.io/projected/e8352713-f7bb-4055-a810-b969caf48a11-kube-api-access-9t2t6\") pod \"maas-api-revoke-sub-f2b8ef01-pzl5h\" (UID: \"e8352713-f7bb-4055-a810-b969caf48a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:46.033295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:46.033260    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b8ef01-pzl5h\" (UID: \"e8352713-f7bb-4055-a810-b969caf48a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:46.033295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:46.033299    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b8ef01-pzl5h\" (UID: \"e8352713-f7bb-4055-a810-b969caf48a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:46.033660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:46.033427    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2t6\" (UniqueName: \"kubernetes.io/projected/e8352713-f7bb-4055-a810-b969caf48a11-kube-api-access-9t2t6\") pod \"maas-api-revoke-sub-f2b8ef01-pzl5h\" (UID: \"e8352713-f7bb-4055-a810-b969caf48a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:46.034144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:46.034122    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b8ef01-pzl5h\" (UID: \"e8352713-f7bb-4055-a810-b969caf48a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:46.035946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:46.035929    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b8ef01-pzl5h\" (UID: \"e8352713-f7bb-4055-a810-b969caf48a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:46.047190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:46.047169    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2t6\" (UniqueName: \"kubernetes.io/projected/e8352713-f7bb-4055-a810-b969caf48a11-kube-api-access-9t2t6\") pod \"maas-api-revoke-sub-f2b8ef01-pzl5h\" (UID: \"e8352713-f7bb-4055-a810-b969caf48a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:46.168455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:46.168356    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:46.455578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:46.455550    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"]
Oct 30 21:23:46.456033 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:23:46.456011    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8352713_f7bb_4055_a810_b969caf48a11.slice/crio-370cdd886a1e275266ad8781828061b7be091f053eef2098e64f4adfd5cf6850 WatchSource:0}: Error finding container 370cdd886a1e275266ad8781828061b7be091f053eef2098e64f4adfd5cf6850: Status 404 returned error can't find the container with id 370cdd886a1e275266ad8781828061b7be091f053eef2098e64f4adfd5cf6850
Oct 30 21:23:46.511877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:46.511843    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h" event={"ID":"e8352713-f7bb-4055-a810-b969caf48a11","Type":"ContainerStarted","Data":"370cdd886a1e275266ad8781828061b7be091f053eef2098e64f4adfd5cf6850"}
Oct 30 21:23:47.574035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:47.573732    4918 generic.go:356] "Generic (PLEG): container finished" podID="e8352713-f7bb-4055-a810-b969caf48a11" containerID="0d8819a292b0267d5bfb7468925962fc59e13292deb7b5a2f7389694ed63e6ce" exitCode=0
Oct 30 21:23:47.574035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:47.573779    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h" event={"ID":"e8352713-f7bb-4055-a810-b969caf48a11","Type":"ContainerDied","Data":"0d8819a292b0267d5bfb7468925962fc59e13292deb7b5a2f7389694ed63e6ce"}
Oct 30 21:23:47.858003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:47.857977    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:23:48.012478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.012445    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-socket\") pod \"020138d9-94c2-4a2f-98a6-f92f25f661ee\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") "
Oct 30 21:23:48.012678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.012572    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-envoy\") pod \"020138d9-94c2-4a2f-98a6-f92f25f661ee\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") "
Oct 30 21:23:48.013090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.012690    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-podinfo\") pod \"020138d9-94c2-4a2f-98a6-f92f25f661ee\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") "
Oct 30 21:23:48.013090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.012720    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-data\") pod \"020138d9-94c2-4a2f-98a6-f92f25f661ee\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") "
Oct 30 21:23:48.013090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.012755    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-socket" pod "020138d9-94c2-4a2f-98a6-f92f25f661ee" (UID: "020138d9-94c2-4a2f-98a6-f92f25f661ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:48.013090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.012818    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-kube-api-access-l9zmj\" (UniqueName: \"kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-kube-api-access-l9zmj\") pod \"020138d9-94c2-4a2f-98a6-f92f25f661ee\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") "
Oct 30 21:23:48.013090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.012911    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-credential-socket\") pod \"020138d9-94c2-4a2f-98a6-f92f25f661ee\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") "
Oct 30 21:23:48.013090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.012985    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020138d9-94c2-4a2f-98a6-f92f25f661ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020138d9-94c2-4a2f-98a6-f92f25f661ee-istiod-ca-cert\") pod \"020138d9-94c2-4a2f-98a6-f92f25f661ee\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") "
Oct 30 21:23:48.013090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.013078    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-token\" (UniqueName: \"kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-token\") pod \"020138d9-94c2-4a2f-98a6-f92f25f661ee\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") "
Oct 30 21:23:48.013574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.013113    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-certs\") pod \"020138d9-94c2-4a2f-98a6-f92f25f661ee\" (UID: \"020138d9-94c2-4a2f-98a6-f92f25f661ee\") "
Oct 30 21:23:48.013574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.013150    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-data" pod "020138d9-94c2-4a2f-98a6-f92f25f661ee" (UID: "020138d9-94c2-4a2f-98a6-f92f25f661ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:48.013574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.013532    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020138d9-94c2-4a2f-98a6-f92f25f661ee-istiod-ca-cert" pod "020138d9-94c2-4a2f-98a6-f92f25f661ee" (UID: "020138d9-94c2-4a2f-98a6-f92f25f661ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:48.013574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.013556    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-certs" pod "020138d9-94c2-4a2f-98a6-f92f25f661ee" (UID: "020138d9-94c2-4a2f-98a6-f92f25f661ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:48.013762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.013575    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-credential-socket" pod "020138d9-94c2-4a2f-98a6-f92f25f661ee" (UID: "020138d9-94c2-4a2f-98a6-f92f25f661ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:48.014006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.013984    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.014143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.014011    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.014143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.014103    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/020138d9-94c2-4a2f-98a6-f92f25f661ee-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.014143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.014123    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.014143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.014137    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.015469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.015442    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-podinfo" pod "020138d9-94c2-4a2f-98a6-f92f25f661ee" (UID: "020138d9-94c2-4a2f-98a6-f92f25f661ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:48.015559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.015476    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-kube-api-access-l9zmj" pod "020138d9-94c2-4a2f-98a6-f92f25f661ee" (UID: "020138d9-94c2-4a2f-98a6-f92f25f661ee"). InnerVolumeSpecName "kube-api-access-l9zmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:48.015714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.015695    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-envoy" pod "020138d9-94c2-4a2f-98a6-f92f25f661ee" (UID: "020138d9-94c2-4a2f-98a6-f92f25f661ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:48.016031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.016015    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-token" pod "020138d9-94c2-4a2f-98a6-f92f25f661ee" (UID: "020138d9-94c2-4a2f-98a6-f92f25f661ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:48.116001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.115929    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.116001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.115957    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9zmj\" (UniqueName: \"kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-kube-api-access-l9zmj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.116001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.115970    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.116001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.115978    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/020138d9-94c2-4a2f-98a6-f92f25f661ee-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:48.586238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.586206    4918 generic.go:356] "Generic (PLEG): container finished" podID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerID="139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f" exitCode=0
Oct 30 21:23:48.586706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.586261    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"
Oct 30 21:23:48.586706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.586292    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" event={"ID":"020138d9-94c2-4a2f-98a6-f92f25f661ee","Type":"ContainerDied","Data":"139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f"}
Oct 30 21:23:48.586706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.586339    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx" event={"ID":"020138d9-94c2-4a2f-98a6-f92f25f661ee","Type":"ContainerDied","Data":"c0aedd3f4ab939cab30b895f17fb61e82b0a66a45e2d5a7b75459c2e697c230e"}
Oct 30 21:23:48.586706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.586362    4918 scope.go:122] "RemoveContainer" containerID="139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f"
Oct 30 21:23:48.597638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.597617    4918 scope.go:122] "RemoveContainer" containerID="66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c"
Oct 30 21:23:48.608286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.608261    4918 scope.go:122] "RemoveContainer" containerID="139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f"
Oct 30 21:23:48.608689 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:48.608664    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f\": container with ID starting with 139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f not found: ID does not exist" containerID="139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f"
Oct 30 21:23:48.608800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.608696    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f"} err="failed to get container status \"139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f\": rpc error: code = NotFound desc = could not find container \"139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f\": container with ID starting with 139c5023d40285acaba9a769b77011b2feaa66eb4c4a01445c330f552f1eb73f not found: ID does not exist"
Oct 30 21:23:48.608800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.608728    4918 scope.go:122] "RemoveContainer" containerID="66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c"
Oct 30 21:23:48.610112 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:48.610084    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c\": container with ID starting with 66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c not found: ID does not exist" containerID="66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c"
Oct 30 21:23:48.610195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.610121    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c"} err="failed to get container status \"66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c\": rpc error: code = NotFound desc = could not find container \"66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c\": container with ID starting with 66f7a9d397a91e47115b4c52373bc3bb53151679bb38ea383d171c653f6ef84c not found: ID does not exist"
Oct 30 21:23:48.613617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.613594    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"]
Oct 30 21:23:48.619679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.619658    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-80e936-openshift-default-cf48cd857-9nzqx"]
Oct 30 21:23:48.767183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.767151    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:48.781981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.781952    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"]
Oct 30 21:23:48.782206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.782180    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy" containerID="cri-o://6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2" gracePeriod=30
Oct 30 21:23:48.953065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.951657    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-xks-ca\") pod \"e8352713-f7bb-4055-a810-b969caf48a11\" (UID: \"e8352713-f7bb-4055-a810-b969caf48a11\") "
Oct 30 21:23:48.953065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.951780    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-openshift-ca\") pod \"e8352713-f7bb-4055-a810-b969caf48a11\" (UID: \"e8352713-f7bb-4055-a810-b969caf48a11\") "
Oct 30 21:23:48.953065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:48.951941    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8352713-f7bb-4055-a810-b969caf48a11-kube-api-access-9t2t6\" (UniqueName: \"kubernetes.io/projected/e8352713-f7bb-4055-a810-b969caf48a11-kube-api-access-9t2t6\") pod \"e8352713-f7bb-4055-a810-b969caf48a11\" (UID: \"e8352713-f7bb-4055-a810-b969caf48a11\") "
Oct 30 21:23:49.002070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.001317    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-openshift-ca" pod "e8352713-f7bb-4055-a810-b969caf48a11" (UID: "e8352713-f7bb-4055-a810-b969caf48a11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:49.002188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.002160    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:49.035416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.031786    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-xks-ca" pod "e8352713-f7bb-4055-a810-b969caf48a11" (UID: "e8352713-f7bb-4055-a810-b969caf48a11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:49.035416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.031915    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8352713-f7bb-4055-a810-b969caf48a11-kube-api-access-9t2t6" pod "e8352713-f7bb-4055-a810-b969caf48a11" (UID: "e8352713-f7bb-4055-a810-b969caf48a11"). InnerVolumeSpecName "kube-api-access-9t2t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:49.104414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.103573    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t2t6\" (UniqueName: \"kubernetes.io/projected/e8352713-f7bb-4055-a810-b969caf48a11-kube-api-access-9t2t6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:49.104414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.103613    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8352713-f7bb-4055-a810-b969caf48a11-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:49.126579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.126299    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" path="/var/lib/kubelet/pods/020138d9-94c2-4a2f-98a6-f92f25f661ee/volumes"
Oct 30 21:23:49.583775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.583739    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"]
Oct 30 21:23:49.584313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.584300    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e8352713-f7bb-4055-a810-b969caf48a11" containerName="revoke-keys"
Oct 30 21:23:49.584362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.584315    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8352713-f7bb-4055-a810-b969caf48a11" containerName="revoke-keys"
Oct 30 21:23:49.584362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.584332    4918 cpu_manager.go:414] "containerMap: removing container" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy"
Oct 30 21:23:49.584362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.584337    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy"
Oct 30 21:23:49.584362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.584348    4918 cpu_manager.go:414] "containerMap: removing container" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy"
Oct 30 21:23:49.584362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.584354    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy"
Oct 30 21:23:49.584542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.584531    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8352713-f7bb-4055-a810-b969caf48a11" containerName="revoke-keys"
Oct 30 21:23:49.584575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.584547    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy"
Oct 30 21:23:49.584575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.584556    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="020138d9-94c2-4a2f-98a6-f92f25f661ee" containerName="istio-proxy"
Oct 30 21:23:49.588433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.588417    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.597817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.597782    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"]
Oct 30 21:23:49.600205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.600177    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-6563b6-openshift-default-dockercfg-6rgpg\""
Oct 30 21:23:49.618405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.618356    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h" event={"ID":"e8352713-f7bb-4055-a810-b969caf48a11","Type":"ContainerDied","Data":"370cdd886a1e275266ad8781828061b7be091f053eef2098e64f4adfd5cf6850"}
Oct 30 21:23:49.618559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.618447    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"
Oct 30 21:23:49.618559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.618480    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="370cdd886a1e275266ad8781828061b7be091f053eef2098e64f4adfd5cf6850"
Oct 30 21:23:49.711759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.711729    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-socket\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.711898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.711794    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxxp\" (UniqueName: \"kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-kube-api-access-zkxxp\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.711898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.711883    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-credential-socket\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.711981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.711923    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-data\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.712049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.712029    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-envoy\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.712102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.712090    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istiod-ca-cert\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.712138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.712112    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-podinfo\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.712173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.712141    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-certs\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.712173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.712158    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-token\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.813605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.813568    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-socket\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.813760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.813616    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxxp\" (UniqueName: \"kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-kube-api-access-zkxxp\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.813760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.813672    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-credential-socket\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.813760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.813708    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-data\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.813760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.813757    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-envoy\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.813951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.813824    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istiod-ca-cert\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.813951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.813849    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-podinfo\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.813951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.813892    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-certs\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.813951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.813921    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-token\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.814251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.814037    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-socket\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.814251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.814238    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-data\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.814378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.814306    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-credential-socket\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.814378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.814344    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-certs\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.814751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.814729    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istiod-ca-cert\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.816269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.816245    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-envoy\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.816660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.816639    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-podinfo\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.822882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.822863    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-token\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.825005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.824979    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxxp\" (UniqueName: \"kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-kube-api-access-zkxxp\") pod \"e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") " pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.918590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.918513    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:49.993656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.993621    4918 patch_prober.go:29] interesting pod/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:23:49.993808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:49.993691    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:23:50.261419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:50.257572    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"]
Oct 30 21:23:50.265221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:50.265188    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:23:50.265342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:50.265272    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:23:50.265342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:50.265317    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:23:50.629835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:50.629743    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" event={"ID":"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9","Type":"ContainerStarted","Data":"3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910"}
Oct 30 21:23:50.629835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:50.629816    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" event={"ID":"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9","Type":"ContainerStarted","Data":"fa94d2bd78879689fc08c0a66a0f8487e4e58186cae1684f104e45336419d9bb"}
Oct 30 21:23:50.919316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:50.919231    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:51.155021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.154968    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" podStartSLOduration=2.154953894 podStartE2EDuration="2.154953894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:50.684839547 +0000 UTC m=+1343.731398693" watchObservedRunningTime="2026-10-30 21:23:51.154953894 +0000 UTC m=+1344.201513038"
Oct 30 21:23:51.156223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.156205    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"]
Oct 30 21:23:51.160488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.160471    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"
Oct 30 21:23:51.167626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.167605    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-18b363\""
Oct 30 21:23:51.173427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.171263    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"]
Oct 30 21:23:51.231766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.231732    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"]
Oct 30 21:23:51.236656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.236636    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"
Oct 30 21:23:51.237340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.237318    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"]
Oct 30 21:23:51.237637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.237591    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-18b363-dockercfg-22jvd\""
Oct 30 21:23:51.241884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.241856    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056b1a05-afe2-4d7a-ae27-bdac204de67e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz\" (UID: \"056b1a05-afe2-4d7a-ae27-bdac204de67e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"
Oct 30 21:23:51.242003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.241926    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-686db\" (UniqueName: \"kubernetes.io/projected/056b1a05-afe2-4d7a-ae27-bdac204de67e-kube-api-access-686db\") pod \"payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz\" (UID: \"056b1a05-afe2-4d7a-ae27-bdac204de67e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"
Oct 30 21:23:51.344341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.344295    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f29202-7112-4ba1-a8c0-1c05259e1863-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f\" (UID: \"b2f29202-7112-4ba1-a8c0-1c05259e1863\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"
Oct 30 21:23:51.344560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.344351    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx7wz\" (UniqueName: \"kubernetes.io/projected/b2f29202-7112-4ba1-a8c0-1c05259e1863-kube-api-access-tx7wz\") pod \"payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f\" (UID: \"b2f29202-7112-4ba1-a8c0-1c05259e1863\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"
Oct 30 21:23:51.344643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.344609    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056b1a05-afe2-4d7a-ae27-bdac204de67e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz\" (UID: \"056b1a05-afe2-4d7a-ae27-bdac204de67e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"
Oct 30 21:23:51.344720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.344675    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-686db\" (UniqueName: \"kubernetes.io/projected/056b1a05-afe2-4d7a-ae27-bdac204de67e-kube-api-access-686db\") pod \"payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz\" (UID: \"056b1a05-afe2-4d7a-ae27-bdac204de67e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"
Oct 30 21:23:51.371419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.368980    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056b1a05-afe2-4d7a-ae27-bdac204de67e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz\" (UID: \"056b1a05-afe2-4d7a-ae27-bdac204de67e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"
Oct 30 21:23:51.402577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.388950    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-686db\" (UniqueName: \"kubernetes.io/projected/056b1a05-afe2-4d7a-ae27-bdac204de67e-kube-api-access-686db\") pod \"payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz\" (UID: \"056b1a05-afe2-4d7a-ae27-bdac204de67e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"
Oct 30 21:23:51.451463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.448569    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f29202-7112-4ba1-a8c0-1c05259e1863-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f\" (UID: \"b2f29202-7112-4ba1-a8c0-1c05259e1863\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"
Oct 30 21:23:51.451463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.448636    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx7wz\" (UniqueName: \"kubernetes.io/projected/b2f29202-7112-4ba1-a8c0-1c05259e1863-kube-api-access-tx7wz\") pod \"payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f\" (UID: \"b2f29202-7112-4ba1-a8c0-1c05259e1863\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"
Oct 30 21:23:51.451463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.449890    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f29202-7112-4ba1-a8c0-1c05259e1863-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f\" (UID: \"b2f29202-7112-4ba1-a8c0-1c05259e1863\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"
Oct 30 21:23:51.476414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.461164    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx7wz\" (UniqueName: \"kubernetes.io/projected/b2f29202-7112-4ba1-a8c0-1c05259e1863-kube-api-access-tx7wz\") pod \"payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f\" (UID: \"b2f29202-7112-4ba1-a8c0-1c05259e1863\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"
Oct 30 21:23:51.480416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.477639    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"
Oct 30 21:23:51.553327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.553295    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"
Oct 30 21:23:51.914741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.914714    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"]
Oct 30 21:23:51.916734 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:23:51.916703    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056b1a05_afe2_4d7a_ae27_bdac204de67e.slice/crio-6c0b9373b239b43f8ca1f5c8d27e86e8d17c06a6da1a0602b35a3f18633d94be WatchSource:0}: Error finding container 6c0b9373b239b43f8ca1f5c8d27e86e8d17c06a6da1a0602b35a3f18633d94be: Status 404 returned error can't find the container with id 6c0b9373b239b43f8ca1f5c8d27e86e8d17c06a6da1a0602b35a3f18633d94be
Oct 30 21:23:51.920673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.920651    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 21:23:51.920768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.920694    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" podUID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 21:23:51.950718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:51.943827    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"]
Oct 30 21:23:52.026774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:52.026741    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"]
Oct 30 21:23:52.030353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:52.030328    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58933cb1-g4skp"]
Oct 30 21:23:52.656876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:52.656836    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f" event={"ID":"b2f29202-7112-4ba1-a8c0-1c05259e1863","Type":"ContainerStarted","Data":"714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14"}
Oct 30 21:23:52.657085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:52.656900    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f" event={"ID":"b2f29202-7112-4ba1-a8c0-1c05259e1863","Type":"ContainerStarted","Data":"28474952ad62261e0e2af96c3c8ac30a3c4f64bd62980effe8951f560cbe0acd"}
Oct 30 21:23:52.657291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:52.657267    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"
Oct 30 21:23:52.659079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:52.659048    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz" event={"ID":"056b1a05-afe2-4d7a-ae27-bdac204de67e","Type":"ContainerStarted","Data":"93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e"}
Oct 30 21:23:52.659219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:52.659133    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"
Oct 30 21:23:52.659219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:52.659176    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz" event={"ID":"056b1a05-afe2-4d7a-ae27-bdac204de67e","Type":"ContainerStarted","Data":"6c0b9373b239b43f8ca1f5c8d27e86e8d17c06a6da1a0602b35a3f18633d94be"}
Oct 30 21:23:52.678205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:52.678167    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f" podStartSLOduration=1.678153993 podStartE2EDuration="1.678153993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:52.676992525 +0000 UTC m=+1345.723551674" watchObservedRunningTime="2026-10-30 21:23:52.678153993 +0000 UTC m=+1345.724713137"
Oct 30 21:23:52.735798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:52.735730    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz" podStartSLOduration=1.7357061169999999 podStartE2EDuration="1.735706117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:52.716735334 +0000 UTC m=+1345.763294480" watchObservedRunningTime="2026-10-30 21:23:52.735706117 +0000 UTC m=+1345.782265264"
Oct 30 21:23:52.926460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:52.922982    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 21:23:52.926460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:52.923044    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" podUID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 21:23:53.078617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.078580    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c40f532-0846-4db6-8af6-fef82a28c5ac" path="/var/lib/kubelet/pods/8c40f532-0846-4db6-8af6-fef82a28c5ac/volumes"
Oct 30 21:23:53.353876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.353812    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"]
Oct 30 21:23:53.365948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.365920    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"]
Oct 30 21:23:53.366160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.365942    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"
Oct 30 21:23:53.433439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.426968    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0fc028-dockercfg-sz4gh\""
Oct 30 21:23:53.433439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.427110    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0fc028\""
Oct 30 21:23:53.461473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.461426    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1103cf6f-4d84-4537-8da8-84fcbee8b32d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c\" (UID: \"1103cf6f-4d84-4537-8da8-84fcbee8b32d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"
Oct 30 21:23:53.461679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.461529    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxs7\" (UniqueName: \"kubernetes.io/projected/1103cf6f-4d84-4537-8da8-84fcbee8b32d-kube-api-access-qfxs7\") pod \"payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c\" (UID: \"1103cf6f-4d84-4537-8da8-84fcbee8b32d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"
Oct 30 21:23:53.571419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.569748    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1103cf6f-4d84-4537-8da8-84fcbee8b32d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c\" (UID: \"1103cf6f-4d84-4537-8da8-84fcbee8b32d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"
Oct 30 21:23:53.571419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.569850    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxs7\" (UniqueName: \"kubernetes.io/projected/1103cf6f-4d84-4537-8da8-84fcbee8b32d-kube-api-access-qfxs7\") pod \"payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c\" (UID: \"1103cf6f-4d84-4537-8da8-84fcbee8b32d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"
Oct 30 21:23:53.571419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.571173    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1103cf6f-4d84-4537-8da8-84fcbee8b32d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c\" (UID: \"1103cf6f-4d84-4537-8da8-84fcbee8b32d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"
Oct 30 21:23:53.653169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.653081    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxs7\" (UniqueName: \"kubernetes.io/projected/1103cf6f-4d84-4537-8da8-84fcbee8b32d-kube-api-access-qfxs7\") pod \"payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c\" (UID: \"1103cf6f-4d84-4537-8da8-84fcbee8b32d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"
Oct 30 21:23:53.705419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.704544    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"
Oct 30 21:23:53.961439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.953999    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 21:23:53.961439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:53.954055    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" podUID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 21:23:54.160144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.159769    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"]
Oct 30 21:23:54.169645 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.169619    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"]
Oct 30 21:23:54.169786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.169711    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:54.173802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.173770    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:23:54.173917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.173767    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:23:54.173917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.173898    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:23:54.174874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.174610    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:23:54.269269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.269233    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vqqp\" (UniqueName: \"kubernetes.io/projected/a4cfd891-06f0-448a-b1cf-4237ea2514a7-kube-api-access-8vqqp\") pod \"maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk\" (UID: \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:54.269679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.269653    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk\" (UID: \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:54.269842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.269820    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk\" (UID: \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:54.344455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.343744    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"]
Oct 30 21:23:54.371712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.371679    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vqqp\" (UniqueName: \"kubernetes.io/projected/a4cfd891-06f0-448a-b1cf-4237ea2514a7-kube-api-access-8vqqp\") pod \"maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk\" (UID: \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:54.371894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.371871    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk\" (UID: \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:54.371991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.371958    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk\" (UID: \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:54.372736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.372713    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk\" (UID: \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:54.375248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.375169    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk\" (UID: \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:54.382238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.382219    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vqqp\" (UniqueName: \"kubernetes.io/projected/a4cfd891-06f0-448a-b1cf-4237ea2514a7-kube-api-access-8vqqp\") pod \"maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk\" (UID: \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:54.467996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.467971    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:23:54.533464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.525204    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:54.576162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.574303    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-podinfo\") pod \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") "
Oct 30 21:23:54.576162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.574367    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-socket\") pod \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") "
Oct 30 21:23:54.576162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.574461    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-token\" (UniqueName: \"kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-token\") pod \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") "
Oct 30 21:23:54.576162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.574584    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istiod-ca-cert\") pod \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") "
Oct 30 21:23:54.576162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.574681    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-kube-api-access-2bkhh\" (UniqueName: \"kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-kube-api-access-2bkhh\") pod \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") "
Oct 30 21:23:54.576162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.574732    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-certs\") pod \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") "
Oct 30 21:23:54.576162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.574790    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-data\") pod \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") "
Oct 30 21:23:54.576162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.574827    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-envoy\") pod \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") "
Oct 30 21:23:54.576162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.574879    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-credential-socket\") pod \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\" (UID: \"81ab1dc4-eb49-4815-8c11-00f9cf59602a\") "
Oct 30 21:23:54.576783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.576500    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-credential-socket" pod "81ab1dc4-eb49-4815-8c11-00f9cf59602a" (UID: "81ab1dc4-eb49-4815-8c11-00f9cf59602a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:54.577698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.577652    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-certs" pod "81ab1dc4-eb49-4815-8c11-00f9cf59602a" (UID: "81ab1dc4-eb49-4815-8c11-00f9cf59602a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:54.577698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.577667    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-data" pod "81ab1dc4-eb49-4815-8c11-00f9cf59602a" (UID: "81ab1dc4-eb49-4815-8c11-00f9cf59602a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:54.577881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.577739    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istiod-ca-cert" pod "81ab1dc4-eb49-4815-8c11-00f9cf59602a" (UID: "81ab1dc4-eb49-4815-8c11-00f9cf59602a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:54.577881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.577783    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-socket" pod "81ab1dc4-eb49-4815-8c11-00f9cf59602a" (UID: "81ab1dc4-eb49-4815-8c11-00f9cf59602a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:54.579129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.579098    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-podinfo" pod "81ab1dc4-eb49-4815-8c11-00f9cf59602a" (UID: "81ab1dc4-eb49-4815-8c11-00f9cf59602a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:23:54.580053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.580028    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-token" pod "81ab1dc4-eb49-4815-8c11-00f9cf59602a" (UID: "81ab1dc4-eb49-4815-8c11-00f9cf59602a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:54.580421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.580345    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-kube-api-access-2bkhh" pod "81ab1dc4-eb49-4815-8c11-00f9cf59602a" (UID: "81ab1dc4-eb49-4815-8c11-00f9cf59602a"). InnerVolumeSpecName "kube-api-access-2bkhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:54.581270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.581241    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-envoy" pod "81ab1dc4-eb49-4815-8c11-00f9cf59602a" (UID: "81ab1dc4-eb49-4815-8c11-00f9cf59602a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:23:54.677208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.677178    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.677208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.677206    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.677208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.677217    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.677512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.677226    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.677512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.677235    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.677512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.677246    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bkhh\" (UniqueName: \"kubernetes.io/projected/81ab1dc4-eb49-4815-8c11-00f9cf59602a-kube-api-access-2bkhh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.677512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.677256    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.677512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.677265    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.677512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.677274    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ab1dc4-eb49-4815-8c11-00f9cf59602a-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:54.767669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.762785    4918 generic.go:356] "Generic (PLEG): container finished" podID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerID="6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2" exitCode=0
Oct 30 21:23:54.767669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.762970    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" event={"ID":"81ab1dc4-eb49-4815-8c11-00f9cf59602a","Type":"ContainerDied","Data":"6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2"}
Oct 30 21:23:54.767669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.763019    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7" event={"ID":"81ab1dc4-eb49-4815-8c11-00f9cf59602a","Type":"ContainerDied","Data":"62121475619b16fd30bdf858a1e05c4685e4a20be04bcd47f7609b55d2fd09ca"}
Oct 30 21:23:54.767669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.763048    4918 scope.go:122] "RemoveContainer" containerID="6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2"
Oct 30 21:23:54.767669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.763223    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"
Oct 30 21:23:54.920496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.920282    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 21:23:54.920496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.920343    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" podUID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 21:23:54.945926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.928201    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c" event={"ID":"1103cf6f-4d84-4537-8da8-84fcbee8b32d","Type":"ContainerStarted","Data":"50540d3446351e2b8b655064076ae872e8cad1407ab64c7f29d6245cc3f0b4e2"}
Oct 30 21:23:54.945926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.928304    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c" event={"ID":"1103cf6f-4d84-4537-8da8-84fcbee8b32d","Type":"ContainerStarted","Data":"7fa778a9fc581fdbb2177800bf1d79134fa953762ae55ff01b790abf7f91d99b"}
Oct 30 21:23:54.949825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:54.949793    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"
Oct 30 21:23:55.055765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:55.055053    4918 scope.go:122] "RemoveContainer" containerID="dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa"
Oct 30 21:23:55.070547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:55.070355    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c" podStartSLOduration=2.070336507 podStartE2EDuration="2.070336507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:55.068653325 +0000 UTC m=+1348.115212471" watchObservedRunningTime="2026-10-30 21:23:55.070336507 +0000 UTC m=+1348.116895653"
Oct 30 21:23:55.070614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:55.070572    4918 scope.go:122] "RemoveContainer" containerID="6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2"
Oct 30 21:23:55.070884 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:55.070856    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2\": container with ID starting with 6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2 not found: ID does not exist" containerID="6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2"
Oct 30 21:23:55.070970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:55.070894    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2"} err="failed to get container status \"6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2\": rpc error: code = NotFound desc = could not find container \"6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2\": container with ID starting with 6f223d857d606acf5c913d18ea06aefb5fd16b725433c4c51ed4345d74af28a2 not found: ID does not exist"
Oct 30 21:23:55.070970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:55.070919    4918 scope.go:122] "RemoveContainer" containerID="dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa"
Oct 30 21:23:55.071357 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:55.071330    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa\": container with ID starting with dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa not found: ID does not exist" containerID="dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa"
Oct 30 21:23:55.071555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:55.071362    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa"} err="failed to get container status \"dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa\": rpc error: code = NotFound desc = could not find container \"dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa\": container with ID starting with dd982b03e11f46dbb11dadaced6102da845d03c255a2bfc1a8ad35c5069461aa not found: ID does not exist"
Oct 30 21:23:55.147421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:55.142436    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"]
Oct 30 21:23:55.150212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:55.150107    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c6998-openshift-default-6fdc5899c-z4zx7"]
Oct 30 21:23:55.215413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:55.212197    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"]
Oct 30 21:23:55.924244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:55.923780    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 21:23:55.924244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:55.923841    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" podUID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 21:23:55.945618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:55.945595    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk" event={"ID":"a4cfd891-06f0-448a-b1cf-4237ea2514a7","Type":"ContainerStarted","Data":"c2b059662d6a2358f1fc5b69cf07b7ebdff9dde71fff409778d4e91b32e8913d"}
Oct 30 21:23:56.919634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:56.919532    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 21:23:56.919634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:56.919594    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" podUID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 21:23:56.957024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:56.956991    4918 generic.go:356] "Generic (PLEG): container finished" podID="a4cfd891-06f0-448a-b1cf-4237ea2514a7" containerID="884ffcce38e6304429c3c3fe983e8504bda55780952af447003bf123b687e1f4" exitCode=0
Oct 30 21:23:56.957225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:56.957119    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk" event={"ID":"a4cfd891-06f0-448a-b1cf-4237ea2514a7","Type":"ContainerDied","Data":"884ffcce38e6304429c3c3fe983e8504bda55780952af447003bf123b687e1f4"}
Oct 30 21:23:57.078591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:57.078560    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" path="/var/lib/kubelet/pods/81ab1dc4-eb49-4815-8c11-00f9cf59602a/volumes"
Oct 30 21:23:57.924071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:57.924045    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:57.924548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:57.924529    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:57.925348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:57.925327    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:23:58.219003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:58.218981    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:58.358329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:58.358295    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-openshift-ca\") pod \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\" (UID: \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\") "
Oct 30 21:23:58.358535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:58.358430    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-xks-ca\") pod \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\" (UID: \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\") "
Oct 30 21:23:58.358634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:58.358601    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cfd891-06f0-448a-b1cf-4237ea2514a7-kube-api-access-8vqqp\" (UniqueName: \"kubernetes.io/projected/a4cfd891-06f0-448a-b1cf-4237ea2514a7-kube-api-access-8vqqp\") pod \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\" (UID: \"a4cfd891-06f0-448a-b1cf-4237ea2514a7\") "
Oct 30 21:23:58.360102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:58.360072    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-openshift-ca" pod "a4cfd891-06f0-448a-b1cf-4237ea2514a7" (UID: "a4cfd891-06f0-448a-b1cf-4237ea2514a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:23:58.363830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:58.363802    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cfd891-06f0-448a-b1cf-4237ea2514a7-kube-api-access-8vqqp" pod "a4cfd891-06f0-448a-b1cf-4237ea2514a7" (UID: "a4cfd891-06f0-448a-b1cf-4237ea2514a7"). InnerVolumeSpecName "kube-api-access-8vqqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:23:58.367036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:58.367012    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-xks-ca" pod "a4cfd891-06f0-448a-b1cf-4237ea2514a7" (UID: "a4cfd891-06f0-448a-b1cf-4237ea2514a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:23:58.460963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:58.460877    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:58.460963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:58.460908    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4cfd891-06f0-448a-b1cf-4237ea2514a7-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:58.460963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:58.460921    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vqqp\" (UniqueName: \"kubernetes.io/projected/a4cfd891-06f0-448a-b1cf-4237ea2514a7-kube-api-access-8vqqp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:23:59.028995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:59.028880    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"
Oct 30 21:23:59.031209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:59.031177    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk" event={"ID":"a4cfd891-06f0-448a-b1cf-4237ea2514a7","Type":"ContainerDied","Data":"c2b059662d6a2358f1fc5b69cf07b7ebdff9dde71fff409778d4e91b32e8913d"}
Oct 30 21:23:59.031331 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:23:59.031264    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2b059662d6a2358f1fc5b69cf07b7ebdff9dde71fff409778d4e91b32e8913d"
Oct 30 21:23:59.162736 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:23:59.162694    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:24:00.536677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.536646    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"]
Oct 30 21:24:00.537238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.537225    4918 cpu_manager.go:414] "containerMap: removing container" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy"
Oct 30 21:24:00.537299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.537239    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy"
Oct 30 21:24:00.537299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.537262    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a4cfd891-06f0-448a-b1cf-4237ea2514a7" containerName="revoke-keys"
Oct 30 21:24:00.537299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.537267    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4cfd891-06f0-448a-b1cf-4237ea2514a7" containerName="revoke-keys"
Oct 30 21:24:00.537299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.537277    4918 cpu_manager.go:414] "containerMap: removing container" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy"
Oct 30 21:24:00.537299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.537282    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy"
Oct 30 21:24:00.537478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.537381    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4cfd891-06f0-448a-b1cf-4237ea2514a7" containerName="revoke-keys"
Oct 30 21:24:00.537478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.537410    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy"
Oct 30 21:24:00.541278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.541249    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:00.545623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.545559    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:24:00.545904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.545881    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:24:00.546019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.545977    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:00.546600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.546571    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:24:00.549649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.549627    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"]
Oct 30 21:24:00.603341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.603314    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59a10a62-ntpm8\" (UID: \"278ba805-4821-4ba9-92be-4c02614d0380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:00.603567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.603545    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59a10a62-ntpm8\" (UID: \"278ba805-4821-4ba9-92be-4c02614d0380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:00.603683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.603667    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvfgq\" (UniqueName: \"kubernetes.io/projected/278ba805-4821-4ba9-92be-4c02614d0380-kube-api-access-gvfgq\") pod \"maas-api-revoke-sub-59a10a62-ntpm8\" (UID: \"278ba805-4821-4ba9-92be-4c02614d0380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:00.705431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.705375    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvfgq\" (UniqueName: \"kubernetes.io/projected/278ba805-4821-4ba9-92be-4c02614d0380-kube-api-access-gvfgq\") pod \"maas-api-revoke-sub-59a10a62-ntpm8\" (UID: \"278ba805-4821-4ba9-92be-4c02614d0380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:00.710596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.710560    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59a10a62-ntpm8\" (UID: \"278ba805-4821-4ba9-92be-4c02614d0380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:00.711186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.711134    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59a10a62-ntpm8\" (UID: \"278ba805-4821-4ba9-92be-4c02614d0380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:00.714234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.714209    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59a10a62-ntpm8\" (UID: \"278ba805-4821-4ba9-92be-4c02614d0380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:00.729456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.726627    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59a10a62-ntpm8\" (UID: \"278ba805-4821-4ba9-92be-4c02614d0380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:00.735735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.735468    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvfgq\" (UniqueName: \"kubernetes.io/projected/278ba805-4821-4ba9-92be-4c02614d0380-kube-api-access-gvfgq\") pod \"maas-api-revoke-sub-59a10a62-ntpm8\" (UID: \"278ba805-4821-4ba9-92be-4c02614d0380\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:00.858004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:00.857939    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:01.144653 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:24:01.144618    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278ba805_4821_4ba9_92be_4c02614d0380.slice/crio-37d4436ce79069b14cf8eb10ec198097f78a950d0e35625a62ad66f3c78af147 WatchSource:0}: Error finding container 37d4436ce79069b14cf8eb10ec198097f78a950d0e35625a62ad66f3c78af147: Status 404 returned error can't find the container with id 37d4436ce79069b14cf8eb10ec198097f78a950d0e35625a62ad66f3c78af147
Oct 30 21:24:01.145599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:01.145575    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"]
Oct 30 21:24:02.055691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:02.055651    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8" event={"ID":"278ba805-4821-4ba9-92be-4c02614d0380","Type":"ContainerStarted","Data":"c48d29846cdd2e27508b5989f4cd33cf501f0f84db5645f9e1176bb551705eea"}
Oct 30 21:24:02.056096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:02.055757    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8" event={"ID":"278ba805-4821-4ba9-92be-4c02614d0380","Type":"ContainerStarted","Data":"37d4436ce79069b14cf8eb10ec198097f78a950d0e35625a62ad66f3c78af147"}
Oct 30 21:24:02.080723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:02.080677    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8" podStartSLOduration=2.080663359 podStartE2EDuration="2.080663359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:02.078982186 +0000 UTC m=+1355.125541326" watchObservedRunningTime="2026-10-30 21:24:02.080663359 +0000 UTC m=+1355.127222504"
Oct 30 21:24:02.473804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:02.471786    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": read tcp 10.132.0.2:33112->10.132.0.183:15021: read: connection reset by peer" start-of-body=
Oct 30 21:24:02.473804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:02.471851    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": read tcp 10.132.0.2:33112->10.132.0.183:15021: read: connection reset by peer"
Oct 30 21:24:03.110038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:03.109998    4918 generic.go:356] "Generic (PLEG): container finished" podID="83c07b1b-9c38-4b55-b203-6c922431639c" containerID="67cb28f4dd506e044b9e9573b13dec93952f377ad4e90878b7d283735915c0b6" exitCode=0
Oct 30 21:24:03.113408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:03.113361    4918 generic.go:356] "Generic (PLEG): container finished" podID="278ba805-4821-4ba9-92be-4c02614d0380" containerID="c48d29846cdd2e27508b5989f4cd33cf501f0f84db5645f9e1176bb551705eea" exitCode=0
Oct 30 21:24:03.130312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:03.129105    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" event={"ID":"83c07b1b-9c38-4b55-b203-6c922431639c","Type":"ContainerDied","Data":"67cb28f4dd506e044b9e9573b13dec93952f377ad4e90878b7d283735915c0b6"}
Oct 30 21:24:03.130312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:03.129170    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8" event={"ID":"278ba805-4821-4ba9-92be-4c02614d0380","Type":"ContainerDied","Data":"c48d29846cdd2e27508b5989f4cd33cf501f0f84db5645f9e1176bb551705eea"}
Oct 30 21:24:03.130312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:03.129680    4918 scope.go:122] "RemoveContainer" containerID="67cb28f4dd506e044b9e9573b13dec93952f377ad4e90878b7d283735915c0b6"
Oct 30 21:24:03.134288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:03.134235    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:03.134539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:03.134517    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:03.134707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:03.134688    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:03.712502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:03.712469    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"
Oct 30 21:24:03.713199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:03.713179    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"
Oct 30 21:24:04.123579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.123541    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" event={"ID":"83c07b1b-9c38-4b55-b203-6c922431639c","Type":"ContainerStarted","Data":"6d2569b0120c5bc743719a3108337bf962c8eebbd9df6167f3c8979d2d54e929"}
Oct 30 21:24:04.354089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.354056    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:04.457205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.457122    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"]
Oct 30 21:24:04.457374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.457323    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9" podUID="9a52b55e-ed84-4a7e-9a72-d001e7e116e7" containerName="payload-processing" containerID="cri-o://70fd3096355550ff7a22d4f32921af418b8e891a3005bbf163dc78e1ebc27ba6" gracePeriod=30
Oct 30 21:24:04.471235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.467881    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"]
Oct 30 21:24:04.471235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.468152    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln" podUID="5da9d34e-0952-4f38-8fe3-ea196e8dc3b9" containerName="payload-pre-processing" containerID="cri-o://7fe6a72555e9965cbf941884b8aea945f3580fcf6fbb701042233e6365ce1113" gracePeriod=30
Oct 30 21:24:04.482813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.482784    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-openshift-ca\") pod \"278ba805-4821-4ba9-92be-4c02614d0380\" (UID: \"278ba805-4821-4ba9-92be-4c02614d0380\") "
Oct 30 21:24:04.483041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.483022    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278ba805-4821-4ba9-92be-4c02614d0380-kube-api-access-gvfgq\" (UniqueName: \"kubernetes.io/projected/278ba805-4821-4ba9-92be-4c02614d0380-kube-api-access-gvfgq\") pod \"278ba805-4821-4ba9-92be-4c02614d0380\" (UID: \"278ba805-4821-4ba9-92be-4c02614d0380\") "
Oct 30 21:24:04.483120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.483078    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-xks-ca\") pod \"278ba805-4821-4ba9-92be-4c02614d0380\" (UID: \"278ba805-4821-4ba9-92be-4c02614d0380\") "
Oct 30 21:24:04.483237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.483181    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-openshift-ca" pod "278ba805-4821-4ba9-92be-4c02614d0380" (UID: "278ba805-4821-4ba9-92be-4c02614d0380"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:04.483710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.483685    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:04.525445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.521621    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278ba805-4821-4ba9-92be-4c02614d0380-kube-api-access-gvfgq" pod "278ba805-4821-4ba9-92be-4c02614d0380" (UID: "278ba805-4821-4ba9-92be-4c02614d0380"). InnerVolumeSpecName "kube-api-access-gvfgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:04.525445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.522162    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-xks-ca" pod "278ba805-4821-4ba9-92be-4c02614d0380" (UID: "278ba805-4821-4ba9-92be-4c02614d0380"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:04.600441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.586047    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvfgq\" (UniqueName: \"kubernetes.io/projected/278ba805-4821-4ba9-92be-4c02614d0380-kube-api-access-gvfgq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:04.600441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.586084    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278ba805-4821-4ba9-92be-4c02614d0380-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:04.745625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.745520    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:24:04.746217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.745915    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 21:24:04.746217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:04.745967    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 21:24:05.004465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.002782    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"]
Oct 30 21:24:05.263498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.263131    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-43f6ab-hzbzk"]
Oct 30 21:24:05.264459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.264437    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"
Oct 30 21:24:05.264976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.264948    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8" event={"ID":"278ba805-4821-4ba9-92be-4c02614d0380","Type":"ContainerDied","Data":"37d4436ce79069b14cf8eb10ec198097f78a950d0e35625a62ad66f3c78af147"}
Oct 30 21:24:05.265127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.265097    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37d4436ce79069b14cf8eb10ec198097f78a950d0e35625a62ad66f3c78af147"
Oct 30 21:24:05.269275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.269249    4918 generic.go:356] "Generic (PLEG): container finished" podID="9a52b55e-ed84-4a7e-9a72-d001e7e116e7" containerID="70fd3096355550ff7a22d4f32921af418b8e891a3005bbf163dc78e1ebc27ba6" exitCode=0
Oct 30 21:24:05.269431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.269359    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9" event={"ID":"9a52b55e-ed84-4a7e-9a72-d001e7e116e7","Type":"ContainerDied","Data":"70fd3096355550ff7a22d4f32921af418b8e891a3005bbf163dc78e1ebc27ba6"}
Oct 30 21:24:05.271870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.271845    4918 generic.go:356] "Generic (PLEG): container finished" podID="5da9d34e-0952-4f38-8fe3-ea196e8dc3b9" containerID="7fe6a72555e9965cbf941884b8aea945f3580fcf6fbb701042233e6365ce1113" exitCode=0
Oct 30 21:24:05.271988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.271876    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"
Oct 30 21:24:05.272053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.272029    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln" event={"ID":"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9","Type":"ContainerDied","Data":"7fe6a72555e9965cbf941884b8aea945f3580fcf6fbb701042233e6365ce1113"}
Oct 30 21:24:05.359318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.359295    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"
Oct 30 21:24:05.450193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.450155    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-plugins-config-volume\") pod \"9a52b55e-ed84-4a7e-9a72-d001e7e116e7\" (UID: \"9a52b55e-ed84-4a7e-9a72-d001e7e116e7\") "
Oct 30 21:24:05.450337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.450255    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-kube-api-access-2vfch\" (UniqueName: \"kubernetes.io/projected/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-kube-api-access-2vfch\") pod \"9a52b55e-ed84-4a7e-9a72-d001e7e116e7\" (UID: \"9a52b55e-ed84-4a7e-9a72-d001e7e116e7\") "
Oct 30 21:24:05.450450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.450382    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-kube-api-access-lt7tl\" (UniqueName: \"kubernetes.io/projected/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-kube-api-access-lt7tl\") pod \"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9\" (UID: \"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9\") "
Oct 30 21:24:05.450598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.450532    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-plugins-config-volume\") pod \"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9\" (UID: \"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9\") "
Oct 30 21:24:05.450675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.450596    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-plugins-config-volume" pod "9a52b55e-ed84-4a7e-9a72-d001e7e116e7" (UID: "9a52b55e-ed84-4a7e-9a72-d001e7e116e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:05.451138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.451091    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-plugins-config-volume" pod "5da9d34e-0952-4f38-8fe3-ea196e8dc3b9" (UID: "5da9d34e-0952-4f38-8fe3-ea196e8dc3b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:05.451240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.451221    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:05.451303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.451243    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:05.452708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.452687    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-kube-api-access-2vfch" pod "9a52b55e-ed84-4a7e-9a72-d001e7e116e7" (UID: "9a52b55e-ed84-4a7e-9a72-d001e7e116e7"). InnerVolumeSpecName "kube-api-access-2vfch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:05.452798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.452757    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-kube-api-access-lt7tl" pod "5da9d34e-0952-4f38-8fe3-ea196e8dc3b9" (UID: "5da9d34e-0952-4f38-8fe3-ea196e8dc3b9"). InnerVolumeSpecName "kube-api-access-lt7tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:05.553099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.553019    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vfch\" (UniqueName: \"kubernetes.io/projected/9a52b55e-ed84-4a7e-9a72-d001e7e116e7-kube-api-access-2vfch\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:05.553099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.553048    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt7tl\" (UniqueName: \"kubernetes.io/projected/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9-kube-api-access-lt7tl\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:05.746319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.746281    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 21:24:05.746602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:05.746346    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 21:24:06.281467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.281440    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"
Oct 30 21:24:06.281921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.281436    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9" event={"ID":"9a52b55e-ed84-4a7e-9a72-d001e7e116e7","Type":"ContainerDied","Data":"b35c04e7a34af9924ca921388d3fda6d835c8ade914c961b8c906e5ba8c6f2d8"}
Oct 30 21:24:06.281921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.281582    4918 scope.go:122] "RemoveContainer" containerID="70fd3096355550ff7a22d4f32921af418b8e891a3005bbf163dc78e1ebc27ba6"
Oct 30 21:24:06.283005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.282983    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"
Oct 30 21:24:06.283092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.283011    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln" event={"ID":"5da9d34e-0952-4f38-8fe3-ea196e8dc3b9","Type":"ContainerDied","Data":"2d8c1bd517206412f8cb37b4fe574daf9185e711a0f3820443557c1dc2dde475"}
Oct 30 21:24:06.318085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.318062    4918 scope.go:122] "RemoveContainer" containerID="7fe6a72555e9965cbf941884b8aea945f3580fcf6fbb701042233e6365ce1113"
Oct 30 21:24:06.337361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.337333    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"]
Oct 30 21:24:06.339686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.339666    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-43f6ab-6b8b6fd9bd-5hwf9"]
Oct 30 21:24:06.349602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.349580    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"]
Oct 30 21:24:06.354481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.354451    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-43f6ab-754d78b96c-zf6ln"]
Oct 30 21:24:06.746018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.745982    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 21:24:06.746210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.746046    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 21:24:06.959558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:06.959528    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"
Oct 30 21:24:07.078894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:07.078817    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da9d34e-0952-4f38-8fe3-ea196e8dc3b9" path="/var/lib/kubelet/pods/5da9d34e-0952-4f38-8fe3-ea196e8dc3b9/volumes"
Oct 30 21:24:07.079278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:07.079265    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a52b55e-ed84-4a7e-9a72-d001e7e116e7" path="/var/lib/kubelet/pods/9a52b55e-ed84-4a7e-9a72-d001e7e116e7/volumes"
Oct 30 21:24:07.079738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:07.079724    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4cfd891-06f0-448a-b1cf-4237ea2514a7" path="/var/lib/kubelet/pods/a4cfd891-06f0-448a-b1cf-4237ea2514a7/volumes"
Oct 30 21:24:07.745301 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:07.745270    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 21:24:07.745752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:07.745339    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 21:24:08.745998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:08.745962    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 30 21:24:08.746406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:08.746017    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 30 21:24:09.756490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:09.752949    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:24:09.756490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:09.754223    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:24:09.756490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:09.754989    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:24:10.662814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.662783    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"]
Oct 30 21:24:10.663614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.663590    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5da9d34e-0952-4f38-8fe3-ea196e8dc3b9" containerName="payload-pre-processing"
Oct 30 21:24:10.663805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.663787    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da9d34e-0952-4f38-8fe3-ea196e8dc3b9" containerName="payload-pre-processing"
Oct 30 21:24:10.663986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.663962    4918 cpu_manager.go:414] "containerMap: removing container" podUID="278ba805-4821-4ba9-92be-4c02614d0380" containerName="revoke-keys"
Oct 30 21:24:10.663986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.663984    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278ba805-4821-4ba9-92be-4c02614d0380" containerName="revoke-keys"
Oct 30 21:24:10.664157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.664007    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9a52b55e-ed84-4a7e-9a72-d001e7e116e7" containerName="payload-processing"
Oct 30 21:24:10.664157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.664015    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a52b55e-ed84-4a7e-9a72-d001e7e116e7" containerName="payload-processing"
Oct 30 21:24:10.664256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.664188    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="278ba805-4821-4ba9-92be-4c02614d0380" containerName="revoke-keys"
Oct 30 21:24:10.664256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.664201    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a52b55e-ed84-4a7e-9a72-d001e7e116e7" containerName="payload-processing"
Oct 30 21:24:10.664256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.664220    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ab1dc4-eb49-4815-8c11-00f9cf59602a" containerName="istio-proxy"
Oct 30 21:24:10.664256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.664233    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da9d34e-0952-4f38-8fe3-ea196e8dc3b9" containerName="payload-pre-processing"
Oct 30 21:24:10.669912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.669891    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"]
Oct 30 21:24:10.670051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.670026    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.696259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.696234    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-fde255-openshift-default-dockercfg-v98cw\""
Oct 30 21:24:10.805715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.805678    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-envoy\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.807228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.807197    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-podinfo\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.807552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.807531    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-certs\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.807857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.807833    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-socket\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.808144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.808112    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-credential-socket\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.808327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.808312    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8r6n\" (UniqueName: \"kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-kube-api-access-x8r6n\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.817630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.817603    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-data\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.817853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.817838    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b9e248-61d3-4cf1-823d-da0343cb6dec-istiod-ca-cert\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.818028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.818015    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-token\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.925298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.925202    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-data\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.925298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.925256    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b9e248-61d3-4cf1-823d-da0343cb6dec-istiod-ca-cert\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.925630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.925309    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-token\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.925630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.925365    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-envoy\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.925630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.925550    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-podinfo\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.925806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.925644    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-certs\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.925806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.925692    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-socket\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.925806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.925764    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-credential-socket\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.925964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.925823    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8r6n\" (UniqueName: \"kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-kube-api-access-x8r6n\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.928382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.927168    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b9e248-61d3-4cf1-823d-da0343cb6dec-istiod-ca-cert\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.928382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.927595    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-data\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.928382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.927826    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-credential-socket\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.928382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.928333    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-certs\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.928983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.928962    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-socket\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.962420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.933215    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-podinfo\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.962420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.956845    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-envoy\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.962420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.958041    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-token\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.969418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.966204    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8r6n\" (UniqueName: \"kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-kube-api-access-x8r6n\") pod \"e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") " pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:10.989320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:10.989288    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:11.284177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.284150    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"]
Oct 30 21:24:11.284913 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:24:11.284884    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b9e248_61d3_4cf1_823d_da0343cb6dec.slice/crio-dc6936a54597eac3b9d7f25a1a88f8d3a294f86f08c410f18ee783dfd5e63a42 WatchSource:0}: Error finding container dc6936a54597eac3b9d7f25a1a88f8d3a294f86f08c410f18ee783dfd5e63a42: Status 404 returned error can't find the container with id dc6936a54597eac3b9d7f25a1a88f8d3a294f86f08c410f18ee783dfd5e63a42
Oct 30 21:24:11.287573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.287545    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:11.287635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.287606    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:11.287670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.287633    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:11.340187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.340151    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"]
Oct 30 21:24:11.344082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.344064    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:11.350658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.350631    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:11.350817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.350676    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:24:11.350817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.350730    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:24:11.350817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.350803    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:24:11.353010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.352987    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"]
Oct 30 21:24:11.362961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.360964    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" event={"ID":"46b9e248-61d3-4cf1-823d-da0343cb6dec","Type":"ContainerStarted","Data":"dc6936a54597eac3b9d7f25a1a88f8d3a294f86f08c410f18ee783dfd5e63a42"}
Oct 30 21:24:11.434558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.434525    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl\" (UID: \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:11.434857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.434828    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl\" (UID: \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:11.435043 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.435026    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn82s\" (UniqueName: \"kubernetes.io/projected/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-kube-api-access-fn82s\") pod \"maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl\" (UID: \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:11.536530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.536442    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn82s\" (UniqueName: \"kubernetes.io/projected/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-kube-api-access-fn82s\") pod \"maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl\" (UID: \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:11.536689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.536543    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl\" (UID: \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:11.536762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.536699    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl\" (UID: \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:11.537226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.537202    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl\" (UID: \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:11.539272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.539254    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl\" (UID: \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:11.546129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.546109    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn82s\" (UniqueName: \"kubernetes.io/projected/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-kube-api-access-fn82s\") pod \"maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl\" (UID: \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:11.661531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:11.661498    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:12.069299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:12.069267    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"]
Oct 30 21:24:12.070004 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:24:12.069976    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fb60e1_b52d_406f_a5d4_5a329fbc7a1f.slice/crio-0812e76640df7f94332e1907c8e092c482dcb2dc9b155e3564966f7ad222e6c7 WatchSource:0}: Error finding container 0812e76640df7f94332e1907c8e092c482dcb2dc9b155e3564966f7ad222e6c7: Status 404 returned error can't find the container with id 0812e76640df7f94332e1907c8e092c482dcb2dc9b155e3564966f7ad222e6c7
Oct 30 21:24:12.415503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:12.412937    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" event={"ID":"46b9e248-61d3-4cf1-823d-da0343cb6dec","Type":"ContainerStarted","Data":"9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6"}
Oct 30 21:24:12.419415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:12.416403    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl" event={"ID":"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f","Type":"ContainerStarted","Data":"0812e76640df7f94332e1907c8e092c482dcb2dc9b155e3564966f7ad222e6c7"}
Oct 30 21:24:12.990358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:12.990328    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:12.995613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:12.995587    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:13.036809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:13.034898    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" podStartSLOduration=3.034844423 podStartE2EDuration="3.034844423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:12.458940265 +0000 UTC m=+1365.505499436" watchObservedRunningTime="2026-10-30 21:24:13.034844423 +0000 UTC m=+1366.081403567"
Oct 30 21:24:13.427753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:13.427663    4918 generic.go:356] "Generic (PLEG): container finished" podID="81fb60e1-b52d-406f-a5d4-5a329fbc7a1f" containerID="513fdcc54d121673b8b02902c566e2ca26b4497ed66ffd937364ee59b494b169" exitCode=0
Oct 30 21:24:13.427753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:13.427735    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl" event={"ID":"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f","Type":"ContainerDied","Data":"513fdcc54d121673b8b02902c566e2ca26b4497ed66ffd937364ee59b494b169"}
Oct 30 21:24:13.428436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:13.428373    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:13.430434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:13.430086    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:13.807694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:13.792026    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"]
Oct 30 21:24:13.861794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:13.861762    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"]
Oct 30 21:24:13.893738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:13.867539    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:14.065108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.063697    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6decbf37-t8vfq\" (UID: \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:14.065108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.063763    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z85jr\" (UniqueName: \"kubernetes.io/projected/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-kube-api-access-z85jr\") pod \"maas-api-revoke-sub-6decbf37-t8vfq\" (UID: \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:14.065108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.063941    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6decbf37-t8vfq\" (UID: \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:14.169855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.169814    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6decbf37-t8vfq\" (UID: \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:14.170080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.169882    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85jr\" (UniqueName: \"kubernetes.io/projected/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-kube-api-access-z85jr\") pod \"maas-api-revoke-sub-6decbf37-t8vfq\" (UID: \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:14.170080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.170041    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6decbf37-t8vfq\" (UID: \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:14.171010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.170821    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6decbf37-t8vfq\" (UID: \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:14.229278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.229100    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85jr\" (UniqueName: \"kubernetes.io/projected/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-kube-api-access-z85jr\") pod \"maas-api-revoke-sub-6decbf37-t8vfq\" (UID: \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:14.256744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.256710    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6decbf37-t8vfq\" (UID: \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:14.391316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.391230    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:14.555471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.555441    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"]
Oct 30 21:24:14.590533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.574715    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.605369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.605324    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"]
Oct 30 21:24:14.682994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.682916    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-7db332-openshift-default-dockercfg-445c8\""
Oct 30 21:24:14.724157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.701703    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjm62\" (UniqueName: \"kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-kube-api-access-hjm62\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.724157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.701855    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-podinfo\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.724157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.701963    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-credential-socket\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.724157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.701995    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-data\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.724157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.702046    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-socket\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.724157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.702085    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-envoy\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.724157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.702109    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-token\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.724157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.702165    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-certs\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.724157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.702291    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d77dde2-5153-40d1-8768-1e84c4a210cd-istiod-ca-cert\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.815415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.814498    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-podinfo\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.815415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.814600    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-credential-socket\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.815415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.814636    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-data\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.815415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.814687    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-socket\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.815415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.814727    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-envoy\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.815415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.814749    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-token\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.815415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.814806    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-certs\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.815415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.814933    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d77dde2-5153-40d1-8768-1e84c4a210cd-istiod-ca-cert\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.815415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.814994    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjm62\" (UniqueName: \"kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-kube-api-access-hjm62\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.816202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.815750    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-certs\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.816272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.816230    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-credential-socket\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.816788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.816373    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d77dde2-5153-40d1-8768-1e84c4a210cd-istiod-ca-cert\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.816788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.816546    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-data\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.817259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.817215    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-socket\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.829479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.821668    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-podinfo\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.829479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.826818    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjm62\" (UniqueName: \"kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-kube-api-access-hjm62\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.829736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.829710    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-envoy\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:14.831010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:14.830985    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-token\") pod \"e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") " pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:15.043417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.040019    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:15.402459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.387892    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"]
Oct 30 21:24:15.402459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.394042    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:15.480855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.480831    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"
Oct 30 21:24:15.480855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.480833    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl" event={"ID":"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f","Type":"ContainerDied","Data":"0812e76640df7f94332e1907c8e092c482dcb2dc9b155e3564966f7ad222e6c7"}
Oct 30 21:24:15.481096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.480877    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0812e76640df7f94332e1907c8e092c482dcb2dc9b155e3564966f7ad222e6c7"
Oct 30 21:24:15.482327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.482294    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq" event={"ID":"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119","Type":"ContainerStarted","Data":"1d4fa4d0eeb94c3700fc8cc49d040e75892ef9985ab7615bc40c884d6dbd4b03"}
Oct 30 21:24:15.484264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.484236    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-kube-api-access-fn82s\" (UniqueName: \"kubernetes.io/projected/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-kube-api-access-fn82s\") pod \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\" (UID: \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\") "
Oct 30 21:24:15.484348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.484286    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-xks-ca\") pod \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\" (UID: \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\") "
Oct 30 21:24:15.484478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.484455    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-openshift-ca\") pod \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\" (UID: \"81fb60e1-b52d-406f-a5d4-5a329fbc7a1f\") "
Oct 30 21:24:15.497890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.485545    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-openshift-ca" pod "81fb60e1-b52d-406f-a5d4-5a329fbc7a1f" (UID: "81fb60e1-b52d-406f-a5d4-5a329fbc7a1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:15.499295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.499261    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-kube-api-access-fn82s" pod "81fb60e1-b52d-406f-a5d4-5a329fbc7a1f" (UID: "81fb60e1-b52d-406f-a5d4-5a329fbc7a1f"). InnerVolumeSpecName "kube-api-access-fn82s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:15.531692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.531654    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-xks-ca" pod "81fb60e1-b52d-406f-a5d4-5a329fbc7a1f" (UID: "81fb60e1-b52d-406f-a5d4-5a329fbc7a1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:15.588338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.588299    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn82s\" (UniqueName: \"kubernetes.io/projected/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-kube-api-access-fn82s\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:15.588857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.588836    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:15.588975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.588961    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:15.722900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.718552    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"]
Oct 30 21:24:15.732374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.732322    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:15.732488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.732434    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:15.732564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:15.732507    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:16.025130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.025099    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"]
Oct 30 21:24:16.030272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.030228    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5518aeae-dgtd9"]
Oct 30 21:24:16.492824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.492794    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" event={"ID":"5d77dde2-5153-40d1-8768-1e84c4a210cd","Type":"ContainerStarted","Data":"a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce"}
Oct 30 21:24:16.492998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.492851    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" event={"ID":"5d77dde2-5153-40d1-8768-1e84c4a210cd","Type":"ContainerStarted","Data":"5957b16a9cc93b210f2d602367ab24d05dc5a62bcc23ead9eda08fe49e5dc03a"}
Oct 30 21:24:16.494947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.494913    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq" event={"ID":"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119","Type":"ContainerStarted","Data":"fc024cbfb2644dbd23e853f4b5eae83c2f386e34336a864c4f947abae50667c0"}
Oct 30 21:24:16.517156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.517099    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" podStartSLOduration=2.517084897 podStartE2EDuration="2.517084897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:16.514158053 +0000 UTC m=+1369.560717227" watchObservedRunningTime="2026-10-30 21:24:16.517084897 +0000 UTC m=+1369.563644041"
Oct 30 21:24:16.534742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.534698    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq" podStartSLOduration=3.534684963 podStartE2EDuration="3.534684963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:16.531656767 +0000 UTC m=+1369.578215912" watchObservedRunningTime="2026-10-30 21:24:16.534684963 +0000 UTC m=+1369.581244108"
Oct 30 21:24:16.924777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.924701    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"]
Oct 30 21:24:16.925421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.925372    4918 cpu_manager.go:414] "containerMap: removing container" podUID="81fb60e1-b52d-406f-a5d4-5a329fbc7a1f" containerName="revoke-keys"
Oct 30 21:24:16.925421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.925405    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81fb60e1-b52d-406f-a5d4-5a329fbc7a1f" containerName="revoke-keys"
Oct 30 21:24:16.925568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.925557    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="81fb60e1-b52d-406f-a5d4-5a329fbc7a1f" containerName="revoke-keys"
Oct 30 21:24:16.929108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.929093    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"
Oct 30 21:24:16.943658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.943639    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fde255-dockercfg-fjhvs\""
Oct 30 21:24:16.944583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.943673    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fde255\""
Oct 30 21:24:16.945479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:16.945458    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"]
Oct 30 21:24:17.010264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.010231    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvw2s\" (UniqueName: \"kubernetes.io/projected/2f00322d-8998-4823-976c-06e4af9f851f-kube-api-access-hvw2s\") pod \"payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg\" (UID: \"2f00322d-8998-4823-976c-06e4af9f851f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"
Oct 30 21:24:17.010559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.010533    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f00322d-8998-4823-976c-06e4af9f851f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg\" (UID: \"2f00322d-8998-4823-976c-06e4af9f851f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"
Oct 30 21:24:17.052219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.049794    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:17.161695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.139947    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvw2s\" (UniqueName: \"kubernetes.io/projected/2f00322d-8998-4823-976c-06e4af9f851f-kube-api-access-hvw2s\") pod \"payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg\" (UID: \"2f00322d-8998-4823-976c-06e4af9f851f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"
Oct 30 21:24:17.161695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.140196    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f00322d-8998-4823-976c-06e4af9f851f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg\" (UID: \"2f00322d-8998-4823-976c-06e4af9f851f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"
Oct 30 21:24:17.161695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.141215    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f00322d-8998-4823-976c-06e4af9f851f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg\" (UID: \"2f00322d-8998-4823-976c-06e4af9f851f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"
Oct 30 21:24:17.203809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.202842    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="624ca571-f68a-45e4-8d12-8a0378147e94" path="/var/lib/kubelet/pods/624ca571-f68a-45e4-8d12-8a0378147e94/volumes"
Oct 30 21:24:17.203809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.203092    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvw2s\" (UniqueName: \"kubernetes.io/projected/2f00322d-8998-4823-976c-06e4af9f851f-kube-api-access-hvw2s\") pod \"payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg\" (UID: \"2f00322d-8998-4823-976c-06e4af9f851f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"
Oct 30 21:24:17.203809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.203433    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:17.248847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.248815    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"
Oct 30 21:24:17.505233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.505209    4918 generic.go:356] "Generic (PLEG): container finished" podID="6fe94b2d-0a3e-4063-b3d5-e8a88ef28119" containerID="fc024cbfb2644dbd23e853f4b5eae83c2f386e34336a864c4f947abae50667c0" exitCode=0
Oct 30 21:24:17.505430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.505284    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq" event={"ID":"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119","Type":"ContainerDied","Data":"fc024cbfb2644dbd23e853f4b5eae83c2f386e34336a864c4f947abae50667c0"}
Oct 30 21:24:17.505749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.505730    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:17.506661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.506642    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:17.573073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:17.572961    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"]
Oct 30 21:24:18.514443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.514365    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg" event={"ID":"2f00322d-8998-4823-976c-06e4af9f851f","Type":"ContainerStarted","Data":"b54f9272cb34819d2a338759a3906dce93c3b368aacb45d4105fc97a3f6998ff"}
Oct 30 21:24:18.514920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.514455    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg" event={"ID":"2f00322d-8998-4823-976c-06e4af9f851f","Type":"ContainerStarted","Data":"6aa30767a44c9b27e8532da0a02038e24edc83ffd534ea6ffa034805d1f5575c"}
Oct 30 21:24:18.514920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.514483    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"
Oct 30 21:24:18.533312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.533264    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg" podStartSLOduration=2.5332507570000002 podStartE2EDuration="2.533250757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:18.531428033 +0000 UTC m=+1371.577987197" watchObservedRunningTime="2026-10-30 21:24:18.533250757 +0000 UTC m=+1371.579809902"
Oct 30 21:24:18.756783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.756758    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:18.871590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.871500    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-kube-api-access-z85jr\" (UniqueName: \"kubernetes.io/projected/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-kube-api-access-z85jr\") pod \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\" (UID: \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\") "
Oct 30 21:24:18.871837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.871693    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-xks-ca\") pod \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\" (UID: \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\") "
Oct 30 21:24:18.871837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.871723    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-openshift-ca\") pod \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\" (UID: \"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119\") "
Oct 30 21:24:18.872249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.872184    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-openshift-ca" pod "6fe94b2d-0a3e-4063-b3d5-e8a88ef28119" (UID: "6fe94b2d-0a3e-4063-b3d5-e8a88ef28119"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:18.872501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.872479    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:18.873846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.873820    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-kube-api-access-z85jr" pod "6fe94b2d-0a3e-4063-b3d5-e8a88ef28119" (UID: "6fe94b2d-0a3e-4063-b3d5-e8a88ef28119"). InnerVolumeSpecName "kube-api-access-z85jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:18.874507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.874486    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-xks-ca" pod "6fe94b2d-0a3e-4063-b3d5-e8a88ef28119" (UID: "6fe94b2d-0a3e-4063-b3d5-e8a88ef28119"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:18.979574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.977687    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z85jr\" (UniqueName: \"kubernetes.io/projected/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-kube-api-access-z85jr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:18.979574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:18.977725    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:19.012414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.007447    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"]
Oct 30 21:24:19.012414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.008383    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6fe94b2d-0a3e-4063-b3d5-e8a88ef28119" containerName="revoke-keys"
Oct 30 21:24:19.012414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.008465    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe94b2d-0a3e-4063-b3d5-e8a88ef28119" containerName="revoke-keys"
Oct 30 21:24:19.012414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.008673    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe94b2d-0a3e-4063-b3d5-e8a88ef28119" containerName="revoke-keys"
Oct 30 21:24:19.022418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.016787    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:19.172374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.172301    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"]
Oct 30 21:24:19.234608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.234541    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z\" (UID: \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:19.234608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.234585    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z\" (UID: \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:19.234839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.234696    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rfwb\" (UniqueName: \"kubernetes.io/projected/d2b94eff-3dcb-4517-b4ef-9de0f932000e-kube-api-access-5rfwb\") pod \"maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z\" (UID: \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:19.336148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.336114    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z\" (UID: \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:19.336148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.336152    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z\" (UID: \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:19.336455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.336182    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rfwb\" (UniqueName: \"kubernetes.io/projected/d2b94eff-3dcb-4517-b4ef-9de0f932000e-kube-api-access-5rfwb\") pod \"maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z\" (UID: \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:19.336970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.336892    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z\" (UID: \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:19.338793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.338769    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z\" (UID: \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:19.346408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.346365    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rfwb\" (UniqueName: \"kubernetes.io/projected/d2b94eff-3dcb-4517-b4ef-9de0f932000e-kube-api-access-5rfwb\") pod \"maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z\" (UID: \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:19.457657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.457567    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:19.530124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.530052    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq" event={"ID":"6fe94b2d-0a3e-4063-b3d5-e8a88ef28119","Type":"ContainerDied","Data":"1d4fa4d0eeb94c3700fc8cc49d040e75892ef9985ab7615bc40c884d6dbd4b03"}
Oct 30 21:24:19.530599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.530162    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d4fa4d0eeb94c3700fc8cc49d040e75892ef9985ab7615bc40c884d6dbd4b03"
Oct 30 21:24:19.530599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.530061    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"
Oct 30 21:24:19.738923 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:24:19.738898    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b94eff_3dcb_4517_b4ef_9de0f932000e.slice/crio-b09851ae50021663bcd866cc5c2d7b40f847822a4dc2e9ef398b184181c6b08b WatchSource:0}: Error finding container b09851ae50021663bcd866cc5c2d7b40f847822a4dc2e9ef398b184181c6b08b: Status 404 returned error can't find the container with id b09851ae50021663bcd866cc5c2d7b40f847822a4dc2e9ef398b184181c6b08b
Oct 30 21:24:19.739160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:19.739138    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"]
Oct 30 21:24:20.547504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:20.547452    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z" event={"ID":"d2b94eff-3dcb-4517-b4ef-9de0f932000e","Type":"ContainerStarted","Data":"056e83beac021fe0b47246449bf8dadc2fe1c63c7a8a0ec585a10ffcc0ae0710"}
Oct 30 21:24:20.547953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:20.547565    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z" event={"ID":"d2b94eff-3dcb-4517-b4ef-9de0f932000e","Type":"ContainerStarted","Data":"b09851ae50021663bcd866cc5c2d7b40f847822a4dc2e9ef398b184181c6b08b"}
Oct 30 21:24:20.573826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:20.573758    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z" podStartSLOduration=2.573742824 podStartE2EDuration="2.573742824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:20.571156507 +0000 UTC m=+1373.617715644" watchObservedRunningTime="2026-10-30 21:24:20.573742824 +0000 UTC m=+1373.620301970"
Oct 30 21:24:21.535028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:21.534993    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"]
Oct 30 21:24:21.558185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:21.558154    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"]
Oct 30 21:24:21.560406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:21.560363    4918 generic.go:356] "Generic (PLEG): container finished" podID="d2b94eff-3dcb-4517-b4ef-9de0f932000e" containerID="056e83beac021fe0b47246449bf8dadc2fe1c63c7a8a0ec585a10ffcc0ae0710" exitCode=0
Oct 30 21:24:21.560602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:21.560562    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z" event={"ID":"d2b94eff-3dcb-4517-b4ef-9de0f932000e","Type":"ContainerDied","Data":"056e83beac021fe0b47246449bf8dadc2fe1c63c7a8a0ec585a10ffcc0ae0710"}
Oct 30 21:24:22.724742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:22.724714    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:22.750780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:22.750749    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-xks-ca\") pod \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\" (UID: \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\") "
Oct 30 21:24:22.750941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:22.750912    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-openshift-ca\") pod \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\" (UID: \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\") "
Oct 30 21:24:22.750998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:22.750987    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b94eff-3dcb-4517-b4ef-9de0f932000e-kube-api-access-5rfwb\" (UniqueName: \"kubernetes.io/projected/d2b94eff-3dcb-4517-b4ef-9de0f932000e-kube-api-access-5rfwb\") pod \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\" (UID: \"d2b94eff-3dcb-4517-b4ef-9de0f932000e\") "
Oct 30 21:24:22.751338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:22.751299    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-openshift-ca" pod "d2b94eff-3dcb-4517-b4ef-9de0f932000e" (UID: "d2b94eff-3dcb-4517-b4ef-9de0f932000e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:22.751613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:22.751538    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:22.753843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:22.753779    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-xks-ca" pod "d2b94eff-3dcb-4517-b4ef-9de0f932000e" (UID: "d2b94eff-3dcb-4517-b4ef-9de0f932000e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:22.753843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:22.753832    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b94eff-3dcb-4517-b4ef-9de0f932000e-kube-api-access-5rfwb" pod "d2b94eff-3dcb-4517-b4ef-9de0f932000e" (UID: "d2b94eff-3dcb-4517-b4ef-9de0f932000e"). InnerVolumeSpecName "kube-api-access-5rfwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:22.856503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:22.853114    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b94eff-3dcb-4517-b4ef-9de0f932000e-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:22.856503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:22.853152    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rfwb\" (UniqueName: \"kubernetes.io/projected/d2b94eff-3dcb-4517-b4ef-9de0f932000e-kube-api-access-5rfwb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:23.577725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:23.577686    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z" event={"ID":"d2b94eff-3dcb-4517-b4ef-9de0f932000e","Type":"ContainerDied","Data":"b09851ae50021663bcd866cc5c2d7b40f847822a4dc2e9ef398b184181c6b08b"}
Oct 30 21:24:23.577725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:23.577712    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"
Oct 30 21:24:23.577725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:23.577734    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09851ae50021663bcd866cc5c2d7b40f847822a4dc2e9ef398b184181c6b08b"
Oct 30 21:24:26.262772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.262739    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"]
Oct 30 21:24:26.263621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.263597    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d2b94eff-3dcb-4517-b4ef-9de0f932000e" containerName="revoke-keys"
Oct 30 21:24:26.263621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.263620    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b94eff-3dcb-4517-b4ef-9de0f932000e" containerName="revoke-keys"
Oct 30 21:24:26.263834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.263820    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b94eff-3dcb-4517-b4ef-9de0f932000e" containerName="revoke-keys"
Oct 30 21:24:26.268643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.268624    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:26.273155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.273133    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:26.273295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.273182    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:24:26.273505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.273486    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:24:26.273611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.273573    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:24:26.276255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.276230    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"]
Oct 30 21:24:26.325706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.325677    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141fb921-zzr7j\" (UID: \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:26.325865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.325728    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141fb921-zzr7j\" (UID: \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:26.326000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.325977    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfkjs\" (UniqueName: \"kubernetes.io/projected/5df5523d-2d07-43ba-84d8-4b6bcd947a80-kube-api-access-gfkjs\") pod \"maas-api-revoke-sub-141fb921-zzr7j\" (UID: \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:26.427502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.427468    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfkjs\" (UniqueName: \"kubernetes.io/projected/5df5523d-2d07-43ba-84d8-4b6bcd947a80-kube-api-access-gfkjs\") pod \"maas-api-revoke-sub-141fb921-zzr7j\" (UID: \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:26.427685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.427512    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141fb921-zzr7j\" (UID: \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:26.427685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.427557    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141fb921-zzr7j\" (UID: \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:26.428213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.428185    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141fb921-zzr7j\" (UID: \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:26.430135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.430112    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141fb921-zzr7j\" (UID: \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:26.436048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.436030    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfkjs\" (UniqueName: \"kubernetes.io/projected/5df5523d-2d07-43ba-84d8-4b6bcd947a80-kube-api-access-gfkjs\") pod \"maas-api-revoke-sub-141fb921-zzr7j\" (UID: \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:26.536110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.536013    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk" podUID="98e21012-166b-4a6d-a762-1c1f2ad9d66b" containerName="payload-processing" containerID="cri-o://b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda" gracePeriod=25
Oct 30 21:24:26.558639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.558608    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq" podUID="ff1aaa42-4b18-4050-84f7-baaa4947b1af" containerName="payload-pre-processing" containerID="cri-o://0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f" gracePeriod=25
Oct 30 21:24:26.586461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.586427    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:26.884878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.884842    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"]
Oct 30 21:24:26.886041 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:24:26.886009    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df5523d_2d07_43ba_84d8_4b6bcd947a80.slice/crio-24b83613508ff10996784f3623026c1759527bca0a914451c7516f9eb21f26ac WatchSource:0}: Error finding container 24b83613508ff10996784f3623026c1759527bca0a914451c7516f9eb21f26ac: Status 404 returned error can't find the container with id 24b83613508ff10996784f3623026c1759527bca0a914451c7516f9eb21f26ac
Oct 30 21:24:26.947772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.947744    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"
Oct 30 21:24:26.980653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:26.980625    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"
Oct 30 21:24:27.037217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.037181    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1aaa42-4b18-4050-84f7-baaa4947b1af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1aaa42-4b18-4050-84f7-baaa4947b1af-plugins-config-volume\") pod \"ff1aaa42-4b18-4050-84f7-baaa4947b1af\" (UID: \"ff1aaa42-4b18-4050-84f7-baaa4947b1af\") "
Oct 30 21:24:27.037405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.037364    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e21012-166b-4a6d-a762-1c1f2ad9d66b-kube-api-access-vfw7t\" (UniqueName: \"kubernetes.io/projected/98e21012-166b-4a6d-a762-1c1f2ad9d66b-kube-api-access-vfw7t\") pod \"98e21012-166b-4a6d-a762-1c1f2ad9d66b\" (UID: \"98e21012-166b-4a6d-a762-1c1f2ad9d66b\") "
Oct 30 21:24:27.037489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.037460    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1aaa42-4b18-4050-84f7-baaa4947b1af-kube-api-access-7cm7g\" (UniqueName: \"kubernetes.io/projected/ff1aaa42-4b18-4050-84f7-baaa4947b1af-kube-api-access-7cm7g\") pod \"ff1aaa42-4b18-4050-84f7-baaa4947b1af\" (UID: \"ff1aaa42-4b18-4050-84f7-baaa4947b1af\") "
Oct 30 21:24:27.037606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.037587    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e21012-166b-4a6d-a762-1c1f2ad9d66b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e21012-166b-4a6d-a762-1c1f2ad9d66b-plugins-config-volume\") pod \"98e21012-166b-4a6d-a762-1c1f2ad9d66b\" (UID: \"98e21012-166b-4a6d-a762-1c1f2ad9d66b\") "
Oct 30 21:24:27.037683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.037633    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1aaa42-4b18-4050-84f7-baaa4947b1af-plugins-config-volume" pod "ff1aaa42-4b18-4050-84f7-baaa4947b1af" (UID: "ff1aaa42-4b18-4050-84f7-baaa4947b1af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:27.038054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.038025    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e21012-166b-4a6d-a762-1c1f2ad9d66b-plugins-config-volume" pod "98e21012-166b-4a6d-a762-1c1f2ad9d66b" (UID: "98e21012-166b-4a6d-a762-1c1f2ad9d66b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:27.038788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.038760    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e21012-166b-4a6d-a762-1c1f2ad9d66b-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:27.038896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.038789    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff1aaa42-4b18-4050-84f7-baaa4947b1af-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:27.040429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.040360    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1aaa42-4b18-4050-84f7-baaa4947b1af-kube-api-access-7cm7g" pod "ff1aaa42-4b18-4050-84f7-baaa4947b1af" (UID: "ff1aaa42-4b18-4050-84f7-baaa4947b1af"). InnerVolumeSpecName "kube-api-access-7cm7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:27.040527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.040447    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e21012-166b-4a6d-a762-1c1f2ad9d66b-kube-api-access-vfw7t" pod "98e21012-166b-4a6d-a762-1c1f2ad9d66b" (UID: "98e21012-166b-4a6d-a762-1c1f2ad9d66b"). InnerVolumeSpecName "kube-api-access-vfw7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:27.140505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.140472    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfw7t\" (UniqueName: \"kubernetes.io/projected/98e21012-166b-4a6d-a762-1c1f2ad9d66b-kube-api-access-vfw7t\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:27.140505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.140505    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cm7g\" (UniqueName: \"kubernetes.io/projected/ff1aaa42-4b18-4050-84f7-baaa4947b1af-kube-api-access-7cm7g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:27.633866 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.633825    4918 generic.go:356] "Generic (PLEG): container finished" podID="98e21012-166b-4a6d-a762-1c1f2ad9d66b" containerID="b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda" exitCode=0
Oct 30 21:24:27.634359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.633894    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"
Oct 30 21:24:27.634359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.633924    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk" event={"ID":"98e21012-166b-4a6d-a762-1c1f2ad9d66b","Type":"ContainerDied","Data":"b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda"}
Oct 30 21:24:27.634359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.633974    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk" event={"ID":"98e21012-166b-4a6d-a762-1c1f2ad9d66b","Type":"ContainerDied","Data":"6ea23508ee57969c095e4c16c9afc7ceac188f0eb1a3a9cee3597efa65cad401"}
Oct 30 21:24:27.634359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.633997    4918 scope.go:122] "RemoveContainer" containerID="b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda"
Oct 30 21:24:27.635580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.635552    4918 generic.go:356] "Generic (PLEG): container finished" podID="ff1aaa42-4b18-4050-84f7-baaa4947b1af" containerID="0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f" exitCode=0
Oct 30 21:24:27.635703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.635607    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"
Oct 30 21:24:27.635703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.635633    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq" event={"ID":"ff1aaa42-4b18-4050-84f7-baaa4947b1af","Type":"ContainerDied","Data":"0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f"}
Oct 30 21:24:27.635703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.635699    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq" event={"ID":"ff1aaa42-4b18-4050-84f7-baaa4947b1af","Type":"ContainerDied","Data":"dbec71f541237af560ea1aa7dca917c444483d145337905f6f1d48fdc89f15fb"}
Oct 30 21:24:27.637673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.637651    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j" event={"ID":"5df5523d-2d07-43ba-84d8-4b6bcd947a80","Type":"ContainerStarted","Data":"81d8034aed5f073d988f60797548c47371d531cf8f16dfe1b123b63e7ad8278c"}
Oct 30 21:24:27.637779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.637691    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j" event={"ID":"5df5523d-2d07-43ba-84d8-4b6bcd947a80","Type":"ContainerStarted","Data":"24b83613508ff10996784f3623026c1759527bca0a914451c7516f9eb21f26ac"}
Oct 30 21:24:27.645861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.645841    4918 scope.go:122] "RemoveContainer" containerID="b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda"
Oct 30 21:24:27.646124 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:24:27.646103    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda\": container with ID starting with b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda not found: ID does not exist" containerID="b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda"
Oct 30 21:24:27.646172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.646132    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda"} err="failed to get container status \"b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda\": rpc error: code = NotFound desc = could not find container \"b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda\": container with ID starting with b10dbfe034c151208dbeff6cdec28f89c02956bf6dfb3adb87e4752e1fcb9bda not found: ID does not exist"
Oct 30 21:24:27.646172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.646149    4918 scope.go:122] "RemoveContainer" containerID="0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f"
Oct 30 21:24:27.655957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.655920    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j" podStartSLOduration=1.65590916 podStartE2EDuration="1.65590916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:27.65448312 +0000 UTC m=+1380.701042290" watchObservedRunningTime="2026-10-30 21:24:27.65590916 +0000 UTC m=+1380.702468304"
Oct 30 21:24:27.656539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.656518    4918 scope.go:122] "RemoveContainer" containerID="0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f"
Oct 30 21:24:27.656821 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:24:27.656802    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f\": container with ID starting with 0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f not found: ID does not exist" containerID="0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f"
Oct 30 21:24:27.656883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.656827    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f"} err="failed to get container status \"0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f\": rpc error: code = NotFound desc = could not find container \"0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f\": container with ID starting with 0ac4700dab9cd5ffd74a27903c467f3f43921ff954284cbbbc23678db2a9f34f not found: ID does not exist"
Oct 30 21:24:27.673610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.673590    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"]
Oct 30 21:24:27.676892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.676873    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-83921a-558f7dd75-4khbq"]
Oct 30 21:24:27.687066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.686726    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"]
Oct 30 21:24:27.690071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:27.690053    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-83921a-5b4474d6c5-2wjpk"]
Oct 30 21:24:28.649039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:28.649009    4918 generic.go:356] "Generic (PLEG): container finished" podID="5df5523d-2d07-43ba-84d8-4b6bcd947a80" containerID="81d8034aed5f073d988f60797548c47371d531cf8f16dfe1b123b63e7ad8278c" exitCode=0
Oct 30 21:24:28.649457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:28.649097    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j" event={"ID":"5df5523d-2d07-43ba-84d8-4b6bcd947a80","Type":"ContainerDied","Data":"81d8034aed5f073d988f60797548c47371d531cf8f16dfe1b123b63e7ad8278c"}
Oct 30 21:24:29.078288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:29.078257    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e21012-166b-4a6d-a762-1c1f2ad9d66b" path="/var/lib/kubelet/pods/98e21012-166b-4a6d-a762-1c1f2ad9d66b/volumes"
Oct 30 21:24:29.078770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:29.078756    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1aaa42-4b18-4050-84f7-baaa4947b1af" path="/var/lib/kubelet/pods/ff1aaa42-4b18-4050-84f7-baaa4947b1af/volumes"
Oct 30 21:24:29.257417 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:24:29.254581    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:24:29.448804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:29.448726    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"]
Oct 30 21:24:29.452196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:29.452170    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-83921a-kt8xl"]
Oct 30 21:24:29.486576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:29.485673    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"]
Oct 30 21:24:29.527785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:29.527758    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"]
Oct 30 21:24:29.539601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:29.539573    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"
Oct 30 21:24:29.676869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:29.676838    4918 scope.go:122] "RemoveContainer" containerID="bf5da1143da4fe56bab4d762e96bf713386f748d2b1e17be3a819da80249f666"
Oct 30 21:24:29.690006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:29.689986    4918 scope.go:122] "RemoveContainer" containerID="b73fbe2a2ee65cef71c0c2fabbbe59e7d5b0fca107154c3763e2b990e74a5cbe"
Oct 30 21:24:29.701315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:29.701293    4918 scope.go:122] "RemoveContainer" containerID="0fba8cd59e648977152949248d69653e3186db71875d3f55b499e71879ad03b8"
Oct 30 21:24:29.711667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:29.711649    4918 scope.go:122] "RemoveContainer" containerID="0b4e3c2c11765172d181ec6d64654565fa269bbb575064b711cf3078d6a115b9"
Oct 30 21:24:29.726530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:29.726508    4918 scope.go:122] "RemoveContainer" containerID="26513dba28d48897afbc705c47653e37f4d0eaef640f296d6b6156e7ef4efc8e"
Oct 30 21:24:30.025922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.025575    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:30.185921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.185891    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-xks-ca\") pod \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\" (UID: \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\") "
Oct 30 21:24:30.186112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.185968    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-openshift-ca\") pod \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\" (UID: \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\") "
Oct 30 21:24:30.186112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.186077    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df5523d-2d07-43ba-84d8-4b6bcd947a80-kube-api-access-gfkjs\" (UniqueName: \"kubernetes.io/projected/5df5523d-2d07-43ba-84d8-4b6bcd947a80-kube-api-access-gfkjs\") pod \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\" (UID: \"5df5523d-2d07-43ba-84d8-4b6bcd947a80\") "
Oct 30 21:24:30.186422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.186365    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-openshift-ca" pod "5df5523d-2d07-43ba-84d8-4b6bcd947a80" (UID: "5df5523d-2d07-43ba-84d8-4b6bcd947a80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:30.186692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.186674    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:30.188535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.188506    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df5523d-2d07-43ba-84d8-4b6bcd947a80-kube-api-access-gfkjs" pod "5df5523d-2d07-43ba-84d8-4b6bcd947a80" (UID: "5df5523d-2d07-43ba-84d8-4b6bcd947a80"). InnerVolumeSpecName "kube-api-access-gfkjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:30.188792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.188768    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-xks-ca" pod "5df5523d-2d07-43ba-84d8-4b6bcd947a80" (UID: "5df5523d-2d07-43ba-84d8-4b6bcd947a80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:30.288163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.288130    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5df5523d-2d07-43ba-84d8-4b6bcd947a80-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:30.288163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.288159    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfkjs\" (UniqueName: \"kubernetes.io/projected/5df5523d-2d07-43ba-84d8-4b6bcd947a80-kube-api-access-gfkjs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:30.666602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.666516    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"
Oct 30 21:24:30.666602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.666553    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j" event={"ID":"5df5523d-2d07-43ba-84d8-4b6bcd947a80","Type":"ContainerDied","Data":"24b83613508ff10996784f3623026c1759527bca0a914451c7516f9eb21f26ac"}
Oct 30 21:24:30.666775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:30.666607    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24b83613508ff10996784f3623026c1759527bca0a914451c7516f9eb21f26ac"
Oct 30 21:24:31.116506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:31.116463    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81fb60e1-b52d-406f-a5d4-5a329fbc7a1f" path="/var/lib/kubelet/pods/81fb60e1-b52d-406f-a5d4-5a329fbc7a1f/volumes"
Oct 30 21:24:34.486338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:34.486285    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm" podUID="df366a2f-ee2a-44c4-9b91-1d89db6d1475" containerName="payload-processing" containerID="cri-o://424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d" gracePeriod=25
Oct 30 21:24:34.528584 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:34.528546    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj" podUID="f8d95733-0e8d-4375-a11d-eec651bb170f" containerName="payload-pre-processing" containerID="cri-o://5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86" gracePeriod=25
Oct 30 21:24:35.861805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.861755    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"]
Oct 30 21:24:35.862358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.862345    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ff1aaa42-4b18-4050-84f7-baaa4947b1af" containerName="payload-pre-processing"
Oct 30 21:24:35.862447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.862360    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1aaa42-4b18-4050-84f7-baaa4947b1af" containerName="payload-pre-processing"
Oct 30 21:24:35.862447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.862403    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5df5523d-2d07-43ba-84d8-4b6bcd947a80" containerName="revoke-keys"
Oct 30 21:24:35.862447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.862409    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df5523d-2d07-43ba-84d8-4b6bcd947a80" containerName="revoke-keys"
Oct 30 21:24:35.862447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.862438    4918 cpu_manager.go:414] "containerMap: removing container" podUID="98e21012-166b-4a6d-a762-1c1f2ad9d66b" containerName="payload-processing"
Oct 30 21:24:35.862447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.862443    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e21012-166b-4a6d-a762-1c1f2ad9d66b" containerName="payload-processing"
Oct 30 21:24:35.862641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.862531    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e21012-166b-4a6d-a762-1c1f2ad9d66b" containerName="payload-processing"
Oct 30 21:24:35.862641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.862542    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1aaa42-4b18-4050-84f7-baaa4947b1af" containerName="payload-pre-processing"
Oct 30 21:24:35.862641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.862551    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df5523d-2d07-43ba-84d8-4b6bcd947a80" containerName="revoke-keys"
Oct 30 21:24:35.867487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.867469    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:35.872607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.872586    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:35.872814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.872791    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:24:35.872814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.872791    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:24:35.872986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.872843    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:24:35.873264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:35.873243    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"]
Oct 30 21:24:36.017469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.017429    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-004c94a0-bp572\" (UID: \"22e408ab-ce1b-40b6-8282-0f12715ff41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:36.018846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.018811    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-004c94a0-bp572\" (UID: \"22e408ab-ce1b-40b6-8282-0f12715ff41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:36.027750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.027710    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smqjx\" (UniqueName: \"kubernetes.io/projected/22e408ab-ce1b-40b6-8282-0f12715ff41c-kube-api-access-smqjx\") pod \"maas-api-revoke-sub-004c94a0-bp572\" (UID: \"22e408ab-ce1b-40b6-8282-0f12715ff41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:36.132504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.129404    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-004c94a0-bp572\" (UID: \"22e408ab-ce1b-40b6-8282-0f12715ff41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:36.132504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.129473    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-004c94a0-bp572\" (UID: \"22e408ab-ce1b-40b6-8282-0f12715ff41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:36.132504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.129592    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smqjx\" (UniqueName: \"kubernetes.io/projected/22e408ab-ce1b-40b6-8282-0f12715ff41c-kube-api-access-smqjx\") pod \"maas-api-revoke-sub-004c94a0-bp572\" (UID: \"22e408ab-ce1b-40b6-8282-0f12715ff41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:36.132504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.130979    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-004c94a0-bp572\" (UID: \"22e408ab-ce1b-40b6-8282-0f12715ff41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:36.154420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.151818    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-004c94a0-bp572\" (UID: \"22e408ab-ce1b-40b6-8282-0f12715ff41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:36.162548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.162507    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smqjx\" (UniqueName: \"kubernetes.io/projected/22e408ab-ce1b-40b6-8282-0f12715ff41c-kube-api-access-smqjx\") pod \"maas-api-revoke-sub-004c94a0-bp572\" (UID: \"22e408ab-ce1b-40b6-8282-0f12715ff41c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:36.247156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.247127    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:36.378120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.378088    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"]
Oct 30 21:24:36.378353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.378327    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy" containerID="cri-o://6d2569b0120c5bc743719a3108337bf962c8eebbd9df6167f3c8979d2d54e929" gracePeriod=30
Oct 30 21:24:36.645341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.645300    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"]
Oct 30 21:24:36.767973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.767945    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572" event={"ID":"22e408ab-ce1b-40b6-8282-0f12715ff41c","Type":"ContainerStarted","Data":"b7a14cef85e0fe79bc650e69efed1e5eddc7ed4c1dc511956e11f5cee0c4b7aa"}
Oct 30 21:24:36.822154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.822120    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"]
Oct 30 21:24:36.827678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.827654    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:36.833890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.833867    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"]
Oct 30 21:24:36.988992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.988909    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brthx\" (UniqueName: \"kubernetes.io/projected/852ea420-85d5-4605-ae42-63f63711842c-kube-api-access-brthx\") pod \"maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt\" (UID: \"852ea420-85d5-4605-ae42-63f63711842c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:36.989411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.989151    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ea420-85d5-4605-ae42-63f63711842c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt\" (UID: \"852ea420-85d5-4605-ae42-63f63711842c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:36.989411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:36.989199    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ea420-85d5-4605-ae42-63f63711842c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt\" (UID: \"852ea420-85d5-4605-ae42-63f63711842c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:37.091741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:37.091171    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ea420-85d5-4605-ae42-63f63711842c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt\" (UID: \"852ea420-85d5-4605-ae42-63f63711842c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:37.091741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:37.091372    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brthx\" (UniqueName: \"kubernetes.io/projected/852ea420-85d5-4605-ae42-63f63711842c-kube-api-access-brthx\") pod \"maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt\" (UID: \"852ea420-85d5-4605-ae42-63f63711842c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:37.091741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:37.091542    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ea420-85d5-4605-ae42-63f63711842c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt\" (UID: \"852ea420-85d5-4605-ae42-63f63711842c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:37.092474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:37.092451    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ea420-85d5-4605-ae42-63f63711842c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt\" (UID: \"852ea420-85d5-4605-ae42-63f63711842c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:37.094804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:37.094782    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ea420-85d5-4605-ae42-63f63711842c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt\" (UID: \"852ea420-85d5-4605-ae42-63f63711842c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:37.099912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:37.099892    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brthx\" (UniqueName: \"kubernetes.io/projected/852ea420-85d5-4605-ae42-63f63711842c-kube-api-access-brthx\") pod \"maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt\" (UID: \"852ea420-85d5-4605-ae42-63f63711842c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:37.144332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:37.144304    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:37.465850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:37.462364    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"]
Oct 30 21:24:37.778132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:37.778101    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt" event={"ID":"852ea420-85d5-4605-ae42-63f63711842c","Type":"ContainerStarted","Data":"83978fef2f0814fca5567999aabb4e626fffc8d122cd9a6802fe2136f825aff0"}
Oct 30 21:24:37.779987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:37.779963    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572" event={"ID":"22e408ab-ce1b-40b6-8282-0f12715ff41c","Type":"ContainerStarted","Data":"636d110aaa712187fc5c6b4f8bab87095f7b928e5a1fcc822dedcea2d3ffa94b"}
Oct 30 21:24:37.800432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:37.800343    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572" podStartSLOduration=2.800330041 podStartE2EDuration="2.800330041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:37.799557438 +0000 UTC m=+1390.846116600" watchObservedRunningTime="2026-10-30 21:24:37.800330041 +0000 UTC m=+1390.846889185"
Oct 30 21:24:38.363602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.363570    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj" podUID="f8d95733-0e8d-4375-a11d-eec651bb170f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:38.364028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.363616    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm" podUID="df366a2f-ee2a-44c4-9b91-1d89db6d1475" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:38.667525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.657342    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"]
Oct 30 21:24:38.690414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.682708    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:38.711551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.710210    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"]
Oct 30 21:24:38.741539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.741508    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dba79e2-6dvwg\" (UID: \"5d93c9ac-2a73-47d3-9304-da0a51852b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:38.741741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.741633    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8l6v\" (UniqueName: \"kubernetes.io/projected/5d93c9ac-2a73-47d3-9304-da0a51852b65-kube-api-access-f8l6v\") pod \"maas-api-revoke-sub-3dba79e2-6dvwg\" (UID: \"5d93c9ac-2a73-47d3-9304-da0a51852b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:38.741831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.741740    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dba79e2-6dvwg\" (UID: \"5d93c9ac-2a73-47d3-9304-da0a51852b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:38.789857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.789826    4918 generic.go:356] "Generic (PLEG): container finished" podID="852ea420-85d5-4605-ae42-63f63711842c" containerID="db10839681c9e8806555311d58bfb790454ea5d5a297aa4902d37f9aa5d5f019" exitCode=0
Oct 30 21:24:38.790022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.789912    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt" event={"ID":"852ea420-85d5-4605-ae42-63f63711842c","Type":"ContainerDied","Data":"db10839681c9e8806555311d58bfb790454ea5d5a297aa4902d37f9aa5d5f019"}
Oct 30 21:24:38.791710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.791684    4918 generic.go:356] "Generic (PLEG): container finished" podID="22e408ab-ce1b-40b6-8282-0f12715ff41c" containerID="636d110aaa712187fc5c6b4f8bab87095f7b928e5a1fcc822dedcea2d3ffa94b" exitCode=0
Oct 30 21:24:38.791829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.791767    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572" event={"ID":"22e408ab-ce1b-40b6-8282-0f12715ff41c","Type":"ContainerDied","Data":"636d110aaa712187fc5c6b4f8bab87095f7b928e5a1fcc822dedcea2d3ffa94b"}
Oct 30 21:24:38.843420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.843377    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dba79e2-6dvwg\" (UID: \"5d93c9ac-2a73-47d3-9304-da0a51852b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:38.843581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.843501    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8l6v\" (UniqueName: \"kubernetes.io/projected/5d93c9ac-2a73-47d3-9304-da0a51852b65-kube-api-access-f8l6v\") pod \"maas-api-revoke-sub-3dba79e2-6dvwg\" (UID: \"5d93c9ac-2a73-47d3-9304-da0a51852b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:38.843624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.843604    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dba79e2-6dvwg\" (UID: \"5d93c9ac-2a73-47d3-9304-da0a51852b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:38.844179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.844159    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dba79e2-6dvwg\" (UID: \"5d93c9ac-2a73-47d3-9304-da0a51852b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:38.845971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.845951    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dba79e2-6dvwg\" (UID: \"5d93c9ac-2a73-47d3-9304-da0a51852b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:38.858592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:38.858571    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8l6v\" (UniqueName: \"kubernetes.io/projected/5d93c9ac-2a73-47d3-9304-da0a51852b65-kube-api-access-f8l6v\") pod \"maas-api-revoke-sub-3dba79e2-6dvwg\" (UID: \"5d93c9ac-2a73-47d3-9304-da0a51852b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:39.072419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:39.064599    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:39.920620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:39.914066    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:24:39.920620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:39.914138    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:24:39.954257 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:39.953616    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"]
Oct 30 21:24:40.184517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.184480    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"]
Oct 30 21:24:40.360182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.360151    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:40.378111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.378072    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-xks-ca\") pod \"22e408ab-ce1b-40b6-8282-0f12715ff41c\" (UID: \"22e408ab-ce1b-40b6-8282-0f12715ff41c\") "
Oct 30 21:24:40.378303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.378153    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-openshift-ca\") pod \"22e408ab-ce1b-40b6-8282-0f12715ff41c\" (UID: \"22e408ab-ce1b-40b6-8282-0f12715ff41c\") "
Oct 30 21:24:40.378303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.378264    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e408ab-ce1b-40b6-8282-0f12715ff41c-kube-api-access-smqjx\" (UniqueName: \"kubernetes.io/projected/22e408ab-ce1b-40b6-8282-0f12715ff41c-kube-api-access-smqjx\") pod \"22e408ab-ce1b-40b6-8282-0f12715ff41c\" (UID: \"22e408ab-ce1b-40b6-8282-0f12715ff41c\") "
Oct 30 21:24:40.378998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.378748    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-openshift-ca" pod "22e408ab-ce1b-40b6-8282-0f12715ff41c" (UID: "22e408ab-ce1b-40b6-8282-0f12715ff41c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:40.379205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.379183    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:40.430863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.429731    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-xks-ca" pod "22e408ab-ce1b-40b6-8282-0f12715ff41c" (UID: "22e408ab-ce1b-40b6-8282-0f12715ff41c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:40.430863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.430566    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e408ab-ce1b-40b6-8282-0f12715ff41c-kube-api-access-smqjx" pod "22e408ab-ce1b-40b6-8282-0f12715ff41c" (UID: "22e408ab-ce1b-40b6-8282-0f12715ff41c"). InnerVolumeSpecName "kube-api-access-smqjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:40.483811 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.482268    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e408ab-ce1b-40b6-8282-0f12715ff41c-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:40.483811 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.482309    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smqjx\" (UniqueName: \"kubernetes.io/projected/22e408ab-ce1b-40b6-8282-0f12715ff41c-kube-api-access-smqjx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:40.533418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.523656    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:40.604444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.586492    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/852ea420-85d5-4605-ae42-63f63711842c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ea420-85d5-4605-ae42-63f63711842c-maas-api-xks-ca\") pod \"852ea420-85d5-4605-ae42-63f63711842c\" (UID: \"852ea420-85d5-4605-ae42-63f63711842c\") "
Oct 30 21:24:40.604444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.586631    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852ea420-85d5-4605-ae42-63f63711842c-kube-api-access-brthx\" (UniqueName: \"kubernetes.io/projected/852ea420-85d5-4605-ae42-63f63711842c-kube-api-access-brthx\") pod \"852ea420-85d5-4605-ae42-63f63711842c\" (UID: \"852ea420-85d5-4605-ae42-63f63711842c\") "
Oct 30 21:24:40.604444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.586811    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852ea420-85d5-4605-ae42-63f63711842c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ea420-85d5-4605-ae42-63f63711842c-maas-api-openshift-ca\") pod \"852ea420-85d5-4605-ae42-63f63711842c\" (UID: \"852ea420-85d5-4605-ae42-63f63711842c\") "
Oct 30 21:24:40.633825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.612591    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852ea420-85d5-4605-ae42-63f63711842c-maas-api-openshift-ca" pod "852ea420-85d5-4605-ae42-63f63711842c" (UID: "852ea420-85d5-4605-ae42-63f63711842c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:40.679770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.675701    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852ea420-85d5-4605-ae42-63f63711842c-kube-api-access-brthx" pod "852ea420-85d5-4605-ae42-63f63711842c" (UID: "852ea420-85d5-4605-ae42-63f63711842c"). InnerVolumeSpecName "kube-api-access-brthx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:40.782441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.773012    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brthx\" (UniqueName: \"kubernetes.io/projected/852ea420-85d5-4605-ae42-63f63711842c-kube-api-access-brthx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:40.782441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:40.773045    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/852ea420-85d5-4605-ae42-63f63711842c-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:41.025414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:41.023181    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572" event={"ID":"22e408ab-ce1b-40b6-8282-0f12715ff41c","Type":"ContainerDied","Data":"b7a14cef85e0fe79bc650e69efed1e5eddc7ed4c1dc511956e11f5cee0c4b7aa"}
Oct 30 21:24:41.025414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:41.023248    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7a14cef85e0fe79bc650e69efed1e5eddc7ed4c1dc511956e11f5cee0c4b7aa"
Oct 30 21:24:41.025414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:41.023330    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"
Oct 30 21:24:41.053477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:41.051244    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt" event={"ID":"852ea420-85d5-4605-ae42-63f63711842c","Type":"ContainerDied","Data":"83978fef2f0814fca5567999aabb4e626fffc8d122cd9a6802fe2136f825aff0"}
Oct 30 21:24:41.053477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:41.051308    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83978fef2f0814fca5567999aabb4e626fffc8d122cd9a6802fe2136f825aff0"
Oct 30 21:24:41.053477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:41.051422    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"
Oct 30 21:24:41.098601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:41.086049    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/852ea420-85d5-4605-ae42-63f63711842c-maas-api-xks-ca" pod "852ea420-85d5-4605-ae42-63f63711842c" (UID: "852ea420-85d5-4605-ae42-63f63711842c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:41.124444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:41.099985    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/852ea420-85d5-4605-ae42-63f63711842c-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:41.399905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:41.398557    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg" event={"ID":"5d93c9ac-2a73-47d3-9304-da0a51852b65","Type":"ContainerStarted","Data":"c8c6f58c500fc6924f9c1609fd6b642d01e8af0120e1303f9f32f887a57fae90"}
Oct 30 21:24:41.480709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:41.479975    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:41.725128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:41.717037    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:24:41.765670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:41.764489    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:42.053745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.010596    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:42.204462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.129695    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d" podUID="ed90e959-7770-4c1e-82fd-3abfd1800ce2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:42.204462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.137666    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:42.204462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.171494    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:42.289513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.274813    4918 generic.go:356] "Generic (PLEG): container finished" podID="83c07b1b-9c38-4b55-b203-6c922431639c" containerID="6d2569b0120c5bc743719a3108337bf962c8eebbd9df6167f3c8979d2d54e929" exitCode=0
Oct 30 21:24:42.307501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.290094    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" event={"ID":"83c07b1b-9c38-4b55-b203-6c922431639c","Type":"ContainerDied","Data":"6d2569b0120c5bc743719a3108337bf962c8eebbd9df6167f3c8979d2d54e929"}
Oct 30 21:24:42.307501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.290182    4918 scope.go:122] "RemoveContainer" containerID="67cb28f4dd506e044b9e9573b13dec93952f377ad4e90878b7d283735915c0b6"
Oct 30 21:24:42.307501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.295442    4918 generic.go:356] "Generic (PLEG): container finished" podID="5d93c9ac-2a73-47d3-9304-da0a51852b65" containerID="5a3bd51aa6adbd82acc874b0184c42f45c31624e87405dff80376ce3473f9b08" exitCode=0
Oct 30 21:24:42.307501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.295635    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg" event={"ID":"5d93c9ac-2a73-47d3-9304-da0a51852b65","Type":"ContainerDied","Data":"5a3bd51aa6adbd82acc874b0184c42f45c31624e87405dff80376ce3473f9b08"}
Oct 30 21:24:42.457666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.454074    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:24:42.549550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.549458    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-credential-socket\") pod \"83c07b1b-9c38-4b55-b203-6c922431639c\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") "
Oct 30 21:24:42.549722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.549558    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-istio-token\" (UniqueName: \"kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-istio-token\") pod \"83c07b1b-9c38-4b55-b203-6c922431639c\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") "
Oct 30 21:24:42.549722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.549598    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c07b1b-9c38-4b55-b203-6c922431639c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c07b1b-9c38-4b55-b203-6c922431639c-istiod-ca-cert\") pod \"83c07b1b-9c38-4b55-b203-6c922431639c\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") "
Oct 30 21:24:42.549874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.549788    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-credential-socket" pod "83c07b1b-9c38-4b55-b203-6c922431639c" (UID: "83c07b1b-9c38-4b55-b203-6c922431639c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:42.550117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.550093    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c07b1b-9c38-4b55-b203-6c922431639c-istiod-ca-cert" pod "83c07b1b-9c38-4b55-b203-6c922431639c" (UID: "83c07b1b-9c38-4b55-b203-6c922431639c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:42.550176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.550162    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-certs\") pod \"83c07b1b-9c38-4b55-b203-6c922431639c\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") "
Oct 30 21:24:42.550231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.550217    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-data\") pod \"83c07b1b-9c38-4b55-b203-6c922431639c\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") "
Oct 30 21:24:42.550294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.550261    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-socket\") pod \"83c07b1b-9c38-4b55-b203-6c922431639c\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") "
Oct 30 21:24:42.550348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.550318    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-kube-api-access-sck88\" (UniqueName: \"kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-kube-api-access-sck88\") pod \"83c07b1b-9c38-4b55-b203-6c922431639c\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") "
Oct 30 21:24:42.550459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.550436    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-data" pod "83c07b1b-9c38-4b55-b203-6c922431639c" (UID: "83c07b1b-9c38-4b55-b203-6c922431639c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:42.550543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.550461    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-envoy\") pod \"83c07b1b-9c38-4b55-b203-6c922431639c\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") "
Oct 30 21:24:42.550543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.550476    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-certs" pod "83c07b1b-9c38-4b55-b203-6c922431639c" (UID: "83c07b1b-9c38-4b55-b203-6c922431639c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:42.550543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.550510    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83c07b1b-9c38-4b55-b203-6c922431639c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c07b1b-9c38-4b55-b203-6c922431639c-istio-podinfo\") pod \"83c07b1b-9c38-4b55-b203-6c922431639c\" (UID: \"83c07b1b-9c38-4b55-b203-6c922431639c\") "
Oct 30 21:24:42.550711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.550642    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-socket" pod "83c07b1b-9c38-4b55-b203-6c922431639c" (UID: "83c07b1b-9c38-4b55-b203-6c922431639c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:42.551094 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.551071    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:42.551094 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.551091    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:42.551478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.551102    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:42.551478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.551110    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83c07b1b-9c38-4b55-b203-6c922431639c-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:42.551478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.551119    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:42.553160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.553138    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-envoy" pod "83c07b1b-9c38-4b55-b203-6c922431639c" (UID: "83c07b1b-9c38-4b55-b203-6c922431639c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:24:42.553259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.553238    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-istio-token" pod "83c07b1b-9c38-4b55-b203-6c922431639c" (UID: "83c07b1b-9c38-4b55-b203-6c922431639c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:42.554737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.554717    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83c07b1b-9c38-4b55-b203-6c922431639c-istio-podinfo" pod "83c07b1b-9c38-4b55-b203-6c922431639c" (UID: "83c07b1b-9c38-4b55-b203-6c922431639c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:24:42.554822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.554782    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-kube-api-access-sck88" pod "83c07b1b-9c38-4b55-b203-6c922431639c" (UID: "83c07b1b-9c38-4b55-b203-6c922431639c"). InnerVolumeSpecName "kube-api-access-sck88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:42.652523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.652491    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:42.652523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.652521    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sck88\" (UniqueName: \"kubernetes.io/projected/83c07b1b-9c38-4b55-b203-6c922431639c-kube-api-access-sck88\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:42.652523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.652532    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83c07b1b-9c38-4b55-b203-6c922431639c-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:42.652776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:42.652540    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83c07b1b-9c38-4b55-b203-6c922431639c-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:43.346172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.345201    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"
Oct 30 21:24:43.346172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.345245    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc" event={"ID":"83c07b1b-9c38-4b55-b203-6c922431639c","Type":"ContainerDied","Data":"8811f2d6f1d42c0204a482e5f9dffe6959486a1f14ec6946e8bad29ee29a3df0"}
Oct 30 21:24:43.346172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.345295    4918 scope.go:122] "RemoveContainer" containerID="6d2569b0120c5bc743719a3108337bf962c8eebbd9df6167f3c8979d2d54e929"
Oct 30 21:24:43.375291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.375256    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"]
Oct 30 21:24:43.471416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.470369    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-83921a-openshift-default-5d88c6854f-6f4nc"]
Oct 30 21:24:43.777326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.777306    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:43.816376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.813352    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-xks-ca\") pod \"5d93c9ac-2a73-47d3-9304-da0a51852b65\" (UID: \"5d93c9ac-2a73-47d3-9304-da0a51852b65\") "
Oct 30 21:24:43.816376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.813468    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-openshift-ca\") pod \"5d93c9ac-2a73-47d3-9304-da0a51852b65\" (UID: \"5d93c9ac-2a73-47d3-9304-da0a51852b65\") "
Oct 30 21:24:43.816376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.813508    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d93c9ac-2a73-47d3-9304-da0a51852b65-kube-api-access-f8l6v\" (UniqueName: \"kubernetes.io/projected/5d93c9ac-2a73-47d3-9304-da0a51852b65-kube-api-access-f8l6v\") pod \"5d93c9ac-2a73-47d3-9304-da0a51852b65\" (UID: \"5d93c9ac-2a73-47d3-9304-da0a51852b65\") "
Oct 30 21:24:43.827412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.826802    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-openshift-ca" pod "5d93c9ac-2a73-47d3-9304-da0a51852b65" (UID: "5d93c9ac-2a73-47d3-9304-da0a51852b65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:43.851482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.851438    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d93c9ac-2a73-47d3-9304-da0a51852b65-kube-api-access-f8l6v" pod "5d93c9ac-2a73-47d3-9304-da0a51852b65" (UID: "5d93c9ac-2a73-47d3-9304-da0a51852b65"). InnerVolumeSpecName "kube-api-access-f8l6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:43.857044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.857022    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-xks-ca" pod "5d93c9ac-2a73-47d3-9304-da0a51852b65" (UID: "5d93c9ac-2a73-47d3-9304-da0a51852b65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:43.936655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.936620    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:43.936958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.936877    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d93c9ac-2a73-47d3-9304-da0a51852b65-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:43.937092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:43.937080    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8l6v\" (UniqueName: \"kubernetes.io/projected/5d93c9ac-2a73-47d3-9304-da0a51852b65-kube-api-access-f8l6v\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:44.398410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:44.397153    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg" event={"ID":"5d93c9ac-2a73-47d3-9304-da0a51852b65","Type":"ContainerDied","Data":"c8c6f58c500fc6924f9c1609fd6b642d01e8af0120e1303f9f32f887a57fae90"}
Oct 30 21:24:44.398410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:44.397222    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8c6f58c500fc6924f9c1609fd6b642d01e8af0120e1303f9f32f887a57fae90"
Oct 30 21:24:44.398410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:44.397292    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"
Oct 30 21:24:45.082691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.079962    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" path="/var/lib/kubelet/pods/83c07b1b-9c38-4b55-b203-6c922431639c/volumes"
Oct 30 21:24:45.184874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.184791    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg" podUID="2f00322d-8998-4823-976c-06e4af9f851f" containerName="payload-pre-processing" containerID="cri-o://b54f9272cb34819d2a338759a3906dce93c3b368aacb45d4105fc97a3f6998ff" gracePeriod=25
Oct 30 21:24:45.449744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.449702    4918 generic.go:356] "Generic (PLEG): container finished" podID="2f00322d-8998-4823-976c-06e4af9f851f" containerID="b54f9272cb34819d2a338759a3906dce93c3b368aacb45d4105fc97a3f6998ff" exitCode=0
Oct 30 21:24:45.450234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.449912    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg" event={"ID":"2f00322d-8998-4823-976c-06e4af9f851f","Type":"ContainerDied","Data":"b54f9272cb34819d2a338759a3906dce93c3b368aacb45d4105fc97a3f6998ff"}
Oct 30 21:24:45.708474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.685832    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"]
Oct 30 21:24:45.731448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.725361    4918 cpu_manager.go:414] "containerMap: removing container" podUID="22e408ab-ce1b-40b6-8282-0f12715ff41c" containerName="revoke-keys"
Oct 30 21:24:45.733072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.733045    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e408ab-ce1b-40b6-8282-0f12715ff41c" containerName="revoke-keys"
Oct 30 21:24:45.733311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.733295    4918 cpu_manager.go:414] "containerMap: removing container" podUID="852ea420-85d5-4605-ae42-63f63711842c" containerName="revoke-keys"
Oct 30 21:24:45.733447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.733432    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852ea420-85d5-4605-ae42-63f63711842c" containerName="revoke-keys"
Oct 30 21:24:45.733566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.733553    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5d93c9ac-2a73-47d3-9304-da0a51852b65" containerName="revoke-keys"
Oct 30 21:24:45.733692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.733678    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d93c9ac-2a73-47d3-9304-da0a51852b65" containerName="revoke-keys"
Oct 30 21:24:45.733824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.733811    4918 cpu_manager.go:414] "containerMap: removing container" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy"
Oct 30 21:24:45.733931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.733918    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy"
Oct 30 21:24:45.734059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.734048    4918 cpu_manager.go:414] "containerMap: removing container" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy"
Oct 30 21:24:45.734140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.734129    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy"
Oct 30 21:24:45.737904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.737885    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy"
Oct 30 21:24:45.738044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.738030    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e408ab-ce1b-40b6-8282-0f12715ff41c" containerName="revoke-keys"
Oct 30 21:24:45.738166 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.738156    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="852ea420-85d5-4605-ae42-63f63711842c" containerName="revoke-keys"
Oct 30 21:24:45.738252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.738244    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d93c9ac-2a73-47d3-9304-da0a51852b65" containerName="revoke-keys"
Oct 30 21:24:45.743079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.743055    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:45.752931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.749735    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-67b92c-openshift-default-dockercfg-l2vwp\""
Oct 30 21:24:45.847417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:45.843860    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"]
Oct 30 21:24:46.079672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.071317    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cdd9a-24cd-4674-b165-a21d9781f974-istiod-ca-cert\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.079672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.071405    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-token\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.079672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.071491    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-podinfo\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.079672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.071598    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-envoy\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.079672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.071636    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-credential-socket\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.079672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.071752    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-certs\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.079672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.071818    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p2f5\" (UniqueName: \"kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-kube-api-access-8p2f5\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.079672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.071882    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-socket\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.079672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.071920    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-data\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.120685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.120322    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"
Oct 30 21:24:46.173092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.173065    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f00322d-8998-4823-976c-06e4af9f851f-kube-api-access-hvw2s\" (UniqueName: \"kubernetes.io/projected/2f00322d-8998-4823-976c-06e4af9f851f-kube-api-access-hvw2s\") pod \"2f00322d-8998-4823-976c-06e4af9f851f\" (UID: \"2f00322d-8998-4823-976c-06e4af9f851f\") "
Oct 30 21:24:46.173299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.173285    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f00322d-8998-4823-976c-06e4af9f851f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f00322d-8998-4823-976c-06e4af9f851f-plugins-config-volume\") pod \"2f00322d-8998-4823-976c-06e4af9f851f\" (UID: \"2f00322d-8998-4823-976c-06e4af9f851f\") "
Oct 30 21:24:46.173754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.173719    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f00322d-8998-4823-976c-06e4af9f851f-plugins-config-volume" pod "2f00322d-8998-4823-976c-06e4af9f851f" (UID: "2f00322d-8998-4823-976c-06e4af9f851f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:46.173898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.173827    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cdd9a-24cd-4674-b165-a21d9781f974-istiod-ca-cert\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.173898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.173879    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-token\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.174040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.173966    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-podinfo\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.174477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.174450    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-envoy\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.174587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.174494    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-credential-socket\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.174646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.174585    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-certs\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.174646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.174639    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p2f5\" (UniqueName: \"kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-kube-api-access-8p2f5\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.174768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.174707    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-socket\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.174768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.174742    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-data\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.174880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.174846    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f00322d-8998-4823-976c-06e4af9f851f-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.175030 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.175004    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-credential-socket\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.175121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.175097    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-data\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.175270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.175247    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cdd9a-24cd-4674-b165-a21d9781f974-istiod-ca-cert\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.175502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.175477    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-certs\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.175781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.175763    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-socket\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.176075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.176056    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f00322d-8998-4823-976c-06e4af9f851f-kube-api-access-hvw2s" pod "2f00322d-8998-4823-976c-06e4af9f851f" (UID: "2f00322d-8998-4823-976c-06e4af9f851f"). InnerVolumeSpecName "kube-api-access-hvw2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:46.178300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.178273    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-envoy\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.178594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.178572    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-podinfo\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.182828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.182804    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-token\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.184034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.184017    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p2f5\" (UniqueName: \"kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-kube-api-access-8p2f5\") pod \"e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") " pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.316736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.311084    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvw2s\" (UniqueName: \"kubernetes.io/projected/2f00322d-8998-4823-976c-06e4af9f851f-kube-api-access-hvw2s\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:46.418774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.418672    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:46.466532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.465481    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"]
Oct 30 21:24:46.466532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.466486    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2f00322d-8998-4823-976c-06e4af9f851f" containerName="payload-pre-processing"
Oct 30 21:24:46.466532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.466506    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f00322d-8998-4823-976c-06e4af9f851f" containerName="payload-pre-processing"
Oct 30 21:24:46.467011 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.466714    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c07b1b-9c38-4b55-b203-6c922431639c" containerName="istio-proxy"
Oct 30 21:24:46.467011 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.466729    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f00322d-8998-4823-976c-06e4af9f851f" containerName="payload-pre-processing"
Oct 30 21:24:46.531653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.531619    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"]
Oct 30 21:24:46.531885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.531763    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.541230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.541176    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg" event={"ID":"2f00322d-8998-4823-976c-06e4af9f851f","Type":"ContainerDied","Data":"6aa30767a44c9b27e8532da0a02038e24edc83ffd534ea6ffa034805d1f5575c"}
Oct 30 21:24:46.541339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.541264    4918 scope.go:122] "RemoveContainer" containerID="b54f9272cb34819d2a338759a3906dce93c3b368aacb45d4105fc97a3f6998ff"
Oct 30 21:24:46.541472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.541457    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"
Oct 30 21:24:46.615475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.614451    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"]
Oct 30 21:24:46.619428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.619378    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fde255-7dbc7d6bd7-mv4wg"]
Oct 30 21:24:46.633358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.629543    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.633358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.629685    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.633358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.629716    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.633358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.629836    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.633358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.629958    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.633358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.629986    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.633358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.630008    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.633358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.630051    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngtqn\" (UniqueName: \"kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-kube-api-access-ngtqn\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.633358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.630088    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.731345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.731309    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.731542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.731365    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.731542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.731526    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.731664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.731600    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngtqn\" (UniqueName: \"kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-kube-api-access-ngtqn\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.731664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.731646    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.731765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.731691    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.731877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.731828    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.731877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.731842    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.731877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.731864    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.732054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.731973    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.732113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.732051    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-data\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.732167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.732106    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-certs\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.732665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.732643    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-credential-socket\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.732928 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.732906    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istiod-ca-cert\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.733973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.733955    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-envoy\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.734354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.734333    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-podinfo\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.740000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.739974    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-token\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.740234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.740215    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngtqn\" (UniqueName: \"kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-kube-api-access-ngtqn\") pod \"e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") " pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:46.848958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.848928    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"]
Oct 30 21:24:46.850546 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:24:46.850502    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63cdd9a_24cd_4674_b165_a21d9781f974.slice/crio-9dbe2cf3dc7375cf5a48514966abd607325efe28fdaf5c5d09eb7d4e123394bc WatchSource:0}: Error finding container 9dbe2cf3dc7375cf5a48514966abd607325efe28fdaf5c5d09eb7d4e123394bc: Status 404 returned error can't find the container with id 9dbe2cf3dc7375cf5a48514966abd607325efe28fdaf5c5d09eb7d4e123394bc
Oct 30 21:24:46.854034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.853996    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:46.854120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.854084    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:46.854174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.854143    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:46.890102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:46.890067    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:47.031029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.030951    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"]
Oct 30 21:24:47.056354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.056315    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"]
Oct 30 21:24:47.099008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.098961    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f00322d-8998-4823-976c-06e4af9f851f" path="/var/lib/kubelet/pods/2f00322d-8998-4823-976c-06e4af9f851f/volumes"
Oct 30 21:24:47.126318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.126275    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"]
Oct 30 21:24:47.132253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.132227    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"
Oct 30 21:24:47.138025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.136803    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fde255\""
Oct 30 21:24:47.138025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.137277    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fde255-dockercfg-jtlpv\""
Oct 30 21:24:47.143714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.143691    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"]
Oct 30 21:24:47.201874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.201413    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"]
Oct 30 21:24:47.210780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.210739    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:47.210886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.210830    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:47.210886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.210873    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:47.242431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.242353    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl94x\" (UniqueName: \"kubernetes.io/projected/6ab512cc-872e-4d6e-a072-9dec56c6b06e-kube-api-access-hl94x\") pod \"payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9\" (UID: \"6ab512cc-872e-4d6e-a072-9dec56c6b06e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"
Oct 30 21:24:47.242564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.242472    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab512cc-872e-4d6e-a072-9dec56c6b06e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9\" (UID: \"6ab512cc-872e-4d6e-a072-9dec56c6b06e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"
Oct 30 21:24:47.345565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.343775    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl94x\" (UniqueName: \"kubernetes.io/projected/6ab512cc-872e-4d6e-a072-9dec56c6b06e-kube-api-access-hl94x\") pod \"payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9\" (UID: \"6ab512cc-872e-4d6e-a072-9dec56c6b06e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"
Oct 30 21:24:47.345565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.343851    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab512cc-872e-4d6e-a072-9dec56c6b06e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9\" (UID: \"6ab512cc-872e-4d6e-a072-9dec56c6b06e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"
Oct 30 21:24:47.345565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.345152    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab512cc-872e-4d6e-a072-9dec56c6b06e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9\" (UID: \"6ab512cc-872e-4d6e-a072-9dec56c6b06e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"
Oct 30 21:24:47.356359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.356331    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl94x\" (UniqueName: \"kubernetes.io/projected/6ab512cc-872e-4d6e-a072-9dec56c6b06e-kube-api-access-hl94x\") pod \"payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9\" (UID: \"6ab512cc-872e-4d6e-a072-9dec56c6b06e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"
Oct 30 21:24:47.587125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.586823    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"
Oct 30 21:24:47.684418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.676453    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" event={"ID":"a63cdd9a-24cd-4674-b165-a21d9781f974","Type":"ContainerStarted","Data":"2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223"}
Oct 30 21:24:47.684418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.676545    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" event={"ID":"a63cdd9a-24cd-4674-b165-a21d9781f974","Type":"ContainerStarted","Data":"9dbe2cf3dc7375cf5a48514966abd607325efe28fdaf5c5d09eb7d4e123394bc"}
Oct 30 21:24:47.706307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.699915    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" event={"ID":"e957b4da-6819-42ee-9afa-e89cef9c4bfd","Type":"ContainerStarted","Data":"a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2"}
Oct 30 21:24:47.706307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.699984    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" event={"ID":"e957b4da-6819-42ee-9afa-e89cef9c4bfd","Type":"ContainerStarted","Data":"d4b791873b98a6bfea8a38c77dd438dc1b492264072b7dcc0f37b0a22be96673"}
Oct 30 21:24:47.718938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.718869    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" podStartSLOduration=2.718849536 podStartE2EDuration="2.718849536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:47.717955309 +0000 UTC m=+1400.764514479" watchObservedRunningTime="2026-10-30 21:24:47.718849536 +0000 UTC m=+1400.765408683"
Oct 30 21:24:47.883028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.882817    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"]
Oct 30 21:24:47.896640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:47.896597    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:48.030466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.022474    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6563b6-hfm2z"]
Oct 30 21:24:48.036867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.036798    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" podStartSLOduration=2.03677341 podStartE2EDuration="2.03677341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:47.756020603 +0000 UTC m=+1400.802579780" watchObservedRunningTime="2026-10-30 21:24:48.03677341 +0000 UTC m=+1401.083332556"
Oct 30 21:24:48.330837 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:24:48.330772    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab512cc_872e_4d6e_a072_9dec56c6b06e.slice/crio-afae62df03f003f437c1ea73b7689ef5ad09b2a17cd6de18704224f904c7ddca WatchSource:0}: Error finding container afae62df03f003f437c1ea73b7689ef5ad09b2a17cd6de18704224f904c7ddca: Status 404 returned error can't find the container with id afae62df03f003f437c1ea73b7689ef5ad09b2a17cd6de18704224f904c7ddca
Oct 30 21:24:48.332170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.332142    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"]
Oct 30 21:24:48.365925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.365896    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj" podUID="f8d95733-0e8d-4375-a11d-eec651bb170f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:48.366043 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.365985    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm" podUID="df366a2f-ee2a-44c4-9b91-1d89db6d1475" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:24:48.419750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.419696    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:48.435756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.428958    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:48.557569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.557527    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:24:48.557756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.557591    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:24:48.725465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.723185    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9" event={"ID":"6ab512cc-872e-4d6e-a072-9dec56c6b06e","Type":"ContainerStarted","Data":"7b932f0b075c699c7073359514128dca4ead1eb04f7138249ce18e28c6b9aa5e"}
Oct 30 21:24:48.725465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.723245    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:48.725465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.723274    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9" event={"ID":"6ab512cc-872e-4d6e-a072-9dec56c6b06e","Type":"ContainerStarted","Data":"afae62df03f003f437c1ea73b7689ef5ad09b2a17cd6de18704224f904c7ddca"}
Oct 30 21:24:48.725465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.724009    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"
Oct 30 21:24:48.726498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.726441    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:24:48.761367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.761313    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9" podStartSLOduration=1.7613004559999998 podStartE2EDuration="1.761300456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:48.75800963 +0000 UTC m=+1401.804568776" watchObservedRunningTime="2026-10-30 21:24:48.761300456 +0000 UTC m=+1401.807859600"
Oct 30 21:24:48.900297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.899976    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:24:48.900297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:48.900038    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" podUID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 21:24:49.131145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:49.131059    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b94eff-3dcb-4517-b4ef-9de0f932000e" path="/var/lib/kubelet/pods/d2b94eff-3dcb-4517-b4ef-9de0f932000e/volumes"
Oct 30 21:24:49.892306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:49.892274    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:24:49.894350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:49.893904    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" podUID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 21:24:50.877477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:50.877439    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"]
Oct 30 21:24:50.892766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:50.892725    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:24:50.893410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:50.893353    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" podUID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 21:24:50.910782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:50.909628    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"]
Oct 30 21:24:50.910782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:50.909771    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:50.914422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:50.913733    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:24:50.915228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:50.915168    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:24:50.934261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:50.932206    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk6dz\" (UniqueName: \"kubernetes.io/projected/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-kube-api-access-tk6dz\") pod \"maas-api-revoke-sub-b0d2696f-8mqmr\" (UID: \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:50.934261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:50.932276    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d2696f-8mqmr\" (UID: \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:50.954140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:50.953853    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d2696f-8mqmr\" (UID: \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:50.984048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:50.979789    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:24:50.984048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:50.979924    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:24:51.095421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.080858    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk6dz\" (UniqueName: \"kubernetes.io/projected/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-kube-api-access-tk6dz\") pod \"maas-api-revoke-sub-b0d2696f-8mqmr\" (UID: \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:51.095421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.080954    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d2696f-8mqmr\" (UID: \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:51.095421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.081218    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d2696f-8mqmr\" (UID: \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:51.120053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.107646    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d2696f-8mqmr\" (UID: \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:51.120053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.107966    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d2696f-8mqmr\" (UID: \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:51.146054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.126461    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk6dz\" (UniqueName: \"kubernetes.io/projected/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-kube-api-access-tk6dz\") pod \"maas-api-revoke-sub-b0d2696f-8mqmr\" (UID: \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:51.246487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.245377    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"]
Oct 30 21:24:51.251204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.251176    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-18b363-t6jjt"]
Oct 30 21:24:51.261086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.261064    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:51.680223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.679774    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"]
Oct 30 21:24:51.912475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.912372    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:24:51.912475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.912443    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" podUID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 21:24:51.978255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:51.978134    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr" event={"ID":"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096","Type":"ContainerStarted","Data":"bfe4bdd007d2acbe2ade5f8d4e2b0457e8ff51c168f61291db3e17e352c760d5"}
Oct 30 21:24:52.031881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.031834    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f" podUID="b2f29202-7112-4ba1-a8c0-1c05259e1863" containerName="payload-processing" containerID="cri-o://714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14" gracePeriod=25
Oct 30 21:24:52.056985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.056936    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz" podUID="056b1a05-afe2-4d7a-ae27-bdac204de67e" containerName="payload-pre-processing" containerID="cri-o://93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e" gracePeriod=25
Oct 30 21:24:52.536164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.535796    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"
Oct 30 21:24:52.665093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.665068    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f29202-7112-4ba1-a8c0-1c05259e1863-kube-api-access-tx7wz\" (UniqueName: \"kubernetes.io/projected/b2f29202-7112-4ba1-a8c0-1c05259e1863-kube-api-access-tx7wz\") pod \"b2f29202-7112-4ba1-a8c0-1c05259e1863\" (UID: \"b2f29202-7112-4ba1-a8c0-1c05259e1863\") "
Oct 30 21:24:52.665329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.665309    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f29202-7112-4ba1-a8c0-1c05259e1863-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f29202-7112-4ba1-a8c0-1c05259e1863-plugins-config-volume\") pod \"b2f29202-7112-4ba1-a8c0-1c05259e1863\" (UID: \"b2f29202-7112-4ba1-a8c0-1c05259e1863\") "
Oct 30 21:24:52.666256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.666228    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f29202-7112-4ba1-a8c0-1c05259e1863-plugins-config-volume" pod "b2f29202-7112-4ba1-a8c0-1c05259e1863" (UID: "b2f29202-7112-4ba1-a8c0-1c05259e1863"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:52.668518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.668490    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"
Oct 30 21:24:52.668611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.668555    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f29202-7112-4ba1-a8c0-1c05259e1863-kube-api-access-tx7wz" pod "b2f29202-7112-4ba1-a8c0-1c05259e1863" (UID: "b2f29202-7112-4ba1-a8c0-1c05259e1863"). InnerVolumeSpecName "kube-api-access-tx7wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:52.766667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.766622    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056b1a05-afe2-4d7a-ae27-bdac204de67e-kube-api-access-686db\" (UniqueName: \"kubernetes.io/projected/056b1a05-afe2-4d7a-ae27-bdac204de67e-kube-api-access-686db\") pod \"056b1a05-afe2-4d7a-ae27-bdac204de67e\" (UID: \"056b1a05-afe2-4d7a-ae27-bdac204de67e\") "
Oct 30 21:24:52.766846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.766759    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056b1a05-afe2-4d7a-ae27-bdac204de67e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056b1a05-afe2-4d7a-ae27-bdac204de67e-plugins-config-volume\") pod \"056b1a05-afe2-4d7a-ae27-bdac204de67e\" (UID: \"056b1a05-afe2-4d7a-ae27-bdac204de67e\") "
Oct 30 21:24:52.767213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.767192    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx7wz\" (UniqueName: \"kubernetes.io/projected/b2f29202-7112-4ba1-a8c0-1c05259e1863-kube-api-access-tx7wz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:52.767307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.767213    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f29202-7112-4ba1-a8c0-1c05259e1863-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:52.767812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.767787    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056b1a05-afe2-4d7a-ae27-bdac204de67e-plugins-config-volume" pod "056b1a05-afe2-4d7a-ae27-bdac204de67e" (UID: "056b1a05-afe2-4d7a-ae27-bdac204de67e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:52.769613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.769590    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056b1a05-afe2-4d7a-ae27-bdac204de67e-kube-api-access-686db" pod "056b1a05-afe2-4d7a-ae27-bdac204de67e" (UID: "056b1a05-afe2-4d7a-ae27-bdac204de67e"). InnerVolumeSpecName "kube-api-access-686db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:52.868919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.868832    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056b1a05-afe2-4d7a-ae27-bdac204de67e-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:52.868919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.868865    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-686db\" (UniqueName: \"kubernetes.io/projected/056b1a05-afe2-4d7a-ae27-bdac204de67e-kube-api-access-686db\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:52.917484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.902837    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:24:52.917484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:52.902908    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" podUID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 21:24:53.011430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.010463    4918 generic.go:356] "Generic (PLEG): container finished" podID="056b1a05-afe2-4d7a-ae27-bdac204de67e" containerID="93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e" exitCode=0
Oct 30 21:24:53.011430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.010705    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz" event={"ID":"056b1a05-afe2-4d7a-ae27-bdac204de67e","Type":"ContainerDied","Data":"93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e"}
Oct 30 21:24:53.011430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.010758    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz" event={"ID":"056b1a05-afe2-4d7a-ae27-bdac204de67e","Type":"ContainerDied","Data":"6c0b9373b239b43f8ca1f5c8d27e86e8d17c06a6da1a0602b35a3f18633d94be"}
Oct 30 21:24:53.011430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.010790    4918 scope.go:122] "RemoveContainer" containerID="93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e"
Oct 30 21:24:53.011430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.010966    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"
Oct 30 21:24:53.019580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.019510    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr" event={"ID":"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096","Type":"ContainerStarted","Data":"ed360326e11369a7f3d74df34f525305ccf4036ae9a81edd3d248064dc9de8c5"}
Oct 30 21:24:53.025513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.023351    4918 generic.go:356] "Generic (PLEG): container finished" podID="b2f29202-7112-4ba1-a8c0-1c05259e1863" containerID="714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14" exitCode=0
Oct 30 21:24:53.025513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.023523    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f" event={"ID":"b2f29202-7112-4ba1-a8c0-1c05259e1863","Type":"ContainerDied","Data":"714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14"}
Oct 30 21:24:53.025513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.023564    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f" event={"ID":"b2f29202-7112-4ba1-a8c0-1c05259e1863","Type":"ContainerDied","Data":"28474952ad62261e0e2af96c3c8ac30a3c4f64bd62980effe8951f560cbe0acd"}
Oct 30 21:24:53.025513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.023636    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"
Oct 30 21:24:53.039919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.039903    4918 scope.go:122] "RemoveContainer" containerID="93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e"
Oct 30 21:24:53.040373 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:24:53.040354    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e\": container with ID starting with 93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e not found: ID does not exist" containerID="93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e"
Oct 30 21:24:53.040464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.040412    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e"} err="failed to get container status \"93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e\": rpc error: code = NotFound desc = could not find container \"93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e\": container with ID starting with 93f35c0a45e64bae38a509f7f9408c5065ec719af612423856fee1fd91b9956e not found: ID does not exist"
Oct 30 21:24:53.040464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.040433    4918 scope.go:122] "RemoveContainer" containerID="714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14"
Oct 30 21:24:53.043140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.043097    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr" podStartSLOduration=3.043083012 podStartE2EDuration="3.043083012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:53.041882603 +0000 UTC m=+1406.088441758" watchObservedRunningTime="2026-10-30 21:24:53.043083012 +0000 UTC m=+1406.089642169"
Oct 30 21:24:53.053404 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.053369    4918 scope.go:122] "RemoveContainer" containerID="714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14"
Oct 30 21:24:53.053779 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:24:53.053760    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14\": container with ID starting with 714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14 not found: ID does not exist" containerID="714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14"
Oct 30 21:24:53.053860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.053785    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14"} err="failed to get container status \"714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14\": rpc error: code = NotFound desc = could not find container \"714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14\": container with ID starting with 714234bf076e21c7e9313b31aeb2fe3667754e3d3f3252d814a1cfe794533f14 not found: ID does not exist"
Oct 30 21:24:53.062892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.062863    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"]
Oct 30 21:24:53.068429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.068407    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-18b363-f4bc69f-glmwz"]
Oct 30 21:24:53.136475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.135058    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056b1a05-afe2-4d7a-ae27-bdac204de67e" path="/var/lib/kubelet/pods/056b1a05-afe2-4d7a-ae27-bdac204de67e/volumes"
Oct 30 21:24:53.136475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.135917    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852ea420-85d5-4605-ae42-63f63711842c" path="/var/lib/kubelet/pods/852ea420-85d5-4605-ae42-63f63711842c/volumes"
Oct 30 21:24:53.136475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.136331    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"]
Oct 30 21:24:53.154353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.138770    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"]
Oct 30 21:24:53.154353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.144003    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-18b363-6664ffcb9f-sps8f"]
Oct 30 21:24:53.898425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.897927    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:24:53.898425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:53.898002    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" podUID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 21:24:54.037550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:54.037149    4918 generic.go:356] "Generic (PLEG): container finished" podID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerID="a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce" exitCode=0
Oct 30 21:24:54.037550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:54.037427    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" event={"ID":"5d77dde2-5153-40d1-8768-1e84c4a210cd","Type":"ContainerDied","Data":"a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce"}
Oct 30 21:24:54.038620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:54.038078    4918 scope.go:122] "RemoveContainer" containerID="a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce"
Oct 30 21:24:54.043961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:54.041429    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:54.043961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:54.041579    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:54.043961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:54.041672    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:54.045874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:54.045847    4918 generic.go:356] "Generic (PLEG): container finished" podID="5139dc2e-8d18-4d96-b8d4-c91d6d9e6096" containerID="ed360326e11369a7f3d74df34f525305ccf4036ae9a81edd3d248064dc9de8c5" exitCode=0
Oct 30 21:24:54.046065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:54.046041    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr" event={"ID":"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096","Type":"ContainerDied","Data":"ed360326e11369a7f3d74df34f525305ccf4036ae9a81edd3d248064dc9de8c5"}
Oct 30 21:24:54.917039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:54.916945    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Oct 30 21:24:54.917337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:54.917309    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" podUID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Oct 30 21:24:55.031993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.031846    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"
Oct 30 21:24:55.062418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.062358    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"
Oct 30 21:24:55.067460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.067431    4918 generic.go:356] "Generic (PLEG): container finished" podID="df366a2f-ee2a-44c4-9b91-1d89db6d1475" containerID="424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d" exitCode=0
Oct 30 21:24:55.067605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.067576    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm" event={"ID":"df366a2f-ee2a-44c4-9b91-1d89db6d1475","Type":"ContainerDied","Data":"424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d"}
Oct 30 21:24:55.067670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.067624    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm" event={"ID":"df366a2f-ee2a-44c4-9b91-1d89db6d1475","Type":"ContainerDied","Data":"7dc709f04f392e96dfb2ce4fe9d3d17c8ca032c16f0d4484b82febb9f43cd5dd"}
Oct 30 21:24:55.067670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.067654    4918 scope.go:122] "RemoveContainer" containerID="424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d"
Oct 30 21:24:55.067808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.067792    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"
Oct 30 21:24:55.070502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.070472    4918 generic.go:356] "Generic (PLEG): container finished" podID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerID="9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6" exitCode=0
Oct 30 21:24:55.120602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.120566    4918 generic.go:356] "Generic (PLEG): container finished" podID="f8d95733-0e8d-4375-a11d-eec651bb170f" containerID="5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86" exitCode=0
Oct 30 21:24:55.121167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.120870    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"
Oct 30 21:24:55.132367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.132283    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f29202-7112-4ba1-a8c0-1c05259e1863" path="/var/lib/kubelet/pods/b2f29202-7112-4ba1-a8c0-1c05259e1863/volumes"
Oct 30 21:24:55.135504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.135380    4918 scope.go:122] "RemoveContainer" containerID="424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d"
Oct 30 21:24:55.145349 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:24:55.145245    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d\": container with ID starting with 424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d not found: ID does not exist" containerID="424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d"
Oct 30 21:24:55.145349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.145283    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d"} err="failed to get container status \"424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d\": rpc error: code = NotFound desc = could not find container \"424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d\": container with ID starting with 424a268338a7971019dc5a0888d3c6bbd8e93131f5edf5b35344e19bfabf316d not found: ID does not exist"
Oct 30 21:24:55.145706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.145666    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" event={"ID":"46b9e248-61d3-4cf1-823d-da0343cb6dec","Type":"ContainerDied","Data":"9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6"}
Oct 30 21:24:55.145815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.145745    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" event={"ID":"5d77dde2-5153-40d1-8768-1e84c4a210cd","Type":"ContainerStarted","Data":"119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7"}
Oct 30 21:24:55.145815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.145808    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj" event={"ID":"f8d95733-0e8d-4375-a11d-eec651bb170f","Type":"ContainerDied","Data":"5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86"}
Oct 30 21:24:55.145938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.145856    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj" event={"ID":"f8d95733-0e8d-4375-a11d-eec651bb170f","Type":"ContainerDied","Data":"278906efec59c89d80b5760b90c1b867dbf043e1ed5aff9a66f1a8185956fa08"}
Oct 30 21:24:55.145938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.145889    4918 scope.go:122] "RemoveContainer" containerID="5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86"
Oct 30 21:24:55.146632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.146610    4918 scope.go:122] "RemoveContainer" containerID="9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6"
Oct 30 21:24:55.149554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.148890    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:55.149554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.148974    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:55.149554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.149018    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:24:55.172462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.172438    4918 scope.go:122] "RemoveContainer" containerID="5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86"
Oct 30 21:24:55.173942 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:24:55.173892    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86\": container with ID starting with 5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86 not found: ID does not exist" containerID="5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86"
Oct 30 21:24:55.174070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.173951    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86"} err="failed to get container status \"5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86\": rpc error: code = NotFound desc = could not find container \"5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86\": container with ID starting with 5323f8f63f70eaa1c38021805a29845e8646b20757e23a06c484541c12320d86 not found: ID does not exist"
Oct 30 21:24:55.180858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.180781    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d95733-0e8d-4375-a11d-eec651bb170f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d95733-0e8d-4375-a11d-eec651bb170f-plugins-config-volume\") pod \"f8d95733-0e8d-4375-a11d-eec651bb170f\" (UID: \"f8d95733-0e8d-4375-a11d-eec651bb170f\") "
Oct 30 21:24:55.180974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.180963    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df366a2f-ee2a-44c4-9b91-1d89db6d1475-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df366a2f-ee2a-44c4-9b91-1d89db6d1475-plugins-config-volume\") pod \"df366a2f-ee2a-44c4-9b91-1d89db6d1475\" (UID: \"df366a2f-ee2a-44c4-9b91-1d89db6d1475\") "
Oct 30 21:24:55.181053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.180992    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d95733-0e8d-4375-a11d-eec651bb170f-kube-api-access-kqw92\" (UniqueName: \"kubernetes.io/projected/f8d95733-0e8d-4375-a11d-eec651bb170f-kube-api-access-kqw92\") pod \"f8d95733-0e8d-4375-a11d-eec651bb170f\" (UID: \"f8d95733-0e8d-4375-a11d-eec651bb170f\") "
Oct 30 21:24:55.181053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.181046    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df366a2f-ee2a-44c4-9b91-1d89db6d1475-kube-api-access-vdt5f\" (UniqueName: \"kubernetes.io/projected/df366a2f-ee2a-44c4-9b91-1d89db6d1475-kube-api-access-vdt5f\") pod \"df366a2f-ee2a-44c4-9b91-1d89db6d1475\" (UID: \"df366a2f-ee2a-44c4-9b91-1d89db6d1475\") "
Oct 30 21:24:55.182365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.182338    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d95733-0e8d-4375-a11d-eec651bb170f-plugins-config-volume" pod "f8d95733-0e8d-4375-a11d-eec651bb170f" (UID: "f8d95733-0e8d-4375-a11d-eec651bb170f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:55.182850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.182825    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df366a2f-ee2a-44c4-9b91-1d89db6d1475-plugins-config-volume" pod "df366a2f-ee2a-44c4-9b91-1d89db6d1475" (UID: "df366a2f-ee2a-44c4-9b91-1d89db6d1475"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:55.227789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.227597    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d95733-0e8d-4375-a11d-eec651bb170f-kube-api-access-kqw92" pod "f8d95733-0e8d-4375-a11d-eec651bb170f" (UID: "f8d95733-0e8d-4375-a11d-eec651bb170f"). InnerVolumeSpecName "kube-api-access-kqw92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:55.228191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.228164    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df366a2f-ee2a-44c4-9b91-1d89db6d1475-kube-api-access-vdt5f" pod "df366a2f-ee2a-44c4-9b91-1d89db6d1475" (UID: "df366a2f-ee2a-44c4-9b91-1d89db6d1475"). InnerVolumeSpecName "kube-api-access-vdt5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:55.312432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.311938    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df366a2f-ee2a-44c4-9b91-1d89db6d1475-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:55.312432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.311977    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqw92\" (UniqueName: \"kubernetes.io/projected/f8d95733-0e8d-4375-a11d-eec651bb170f-kube-api-access-kqw92\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:55.312432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.311993    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdt5f\" (UniqueName: \"kubernetes.io/projected/df366a2f-ee2a-44c4-9b91-1d89db6d1475-kube-api-access-vdt5f\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:55.312432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.312008    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d95733-0e8d-4375-a11d-eec651bb170f-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:55.450320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.450282    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"]
Oct 30 21:24:55.464511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.457380    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6563b6-7dd98b99c5-922qm"]
Oct 30 21:24:55.509416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.495590    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:55.548416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.547463    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"]
Oct 30 21:24:55.569623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.564689    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-openshift-ca\") pod \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\" (UID: \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\") "
Oct 30 21:24:55.569623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.564778    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-kube-api-access-tk6dz\" (UniqueName: \"kubernetes.io/projected/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-kube-api-access-tk6dz\") pod \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\" (UID: \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\") "
Oct 30 21:24:55.569623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.564837    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-xks-ca\") pod \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\" (UID: \"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096\") "
Oct 30 21:24:55.569623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.567572    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-openshift-ca" pod "5139dc2e-8d18-4d96-b8d4-c91d6d9e6096" (UID: "5139dc2e-8d18-4d96-b8d4-c91d6d9e6096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:24:55.576415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.570735    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-kube-api-access-tk6dz" pod "5139dc2e-8d18-4d96-b8d4-c91d6d9e6096" (UID: "5139dc2e-8d18-4d96-b8d4-c91d6d9e6096"). InnerVolumeSpecName "kube-api-access-tk6dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:24:55.576415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.572821    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6563b6-b57856c4c-tmzxj"]
Oct 30 21:24:55.587459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.583552    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-xks-ca" pod "5139dc2e-8d18-4d96-b8d4-c91d6d9e6096" (UID: "5139dc2e-8d18-4d96-b8d4-c91d6d9e6096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:24:55.681943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.667430    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:55.681943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.667467    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk6dz\" (UniqueName: \"kubernetes.io/projected/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-kube-api-access-tk6dz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:55.681943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:55.667485    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:24:56.040733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:56.040695    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:24:56.040970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:56.040951    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 21:24:56.041017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:56.040998    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 21:24:56.131775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:56.131021    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" event={"ID":"46b9e248-61d3-4cf1-823d-da0343cb6dec","Type":"ContainerStarted","Data":"71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419"}
Oct 30 21:24:56.135122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:56.135088    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr" event={"ID":"5139dc2e-8d18-4d96-b8d4-c91d6d9e6096","Type":"ContainerDied","Data":"bfe4bdd007d2acbe2ade5f8d4e2b0457e8ff51c168f61291db3e17e352c760d5"}
Oct 30 21:24:56.135278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:56.135148    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfe4bdd007d2acbe2ade5f8d4e2b0457e8ff51c168f61291db3e17e352c760d5"
Oct 30 21:24:56.135278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:56.135220    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"
Oct 30 21:24:56.355780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:56.355695    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:56.357968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:56.357889    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:56.361552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:56.361478    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:24:57.027417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:57.026883    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:57.038417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:57.035528    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:24:57.038417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:57.035584    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 21:24:57.041023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:57.040993    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 21:24:57.041170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:57.041034    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 21:24:57.239471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:57.232655    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df366a2f-ee2a-44c4-9b91-1d89db6d1475" path="/var/lib/kubelet/pods/df366a2f-ee2a-44c4-9b91-1d89db6d1475/volumes"
Oct 30 21:24:57.239471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:57.233210    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d95733-0e8d-4375-a11d-eec651bb170f" path="/var/lib/kubelet/pods/f8d95733-0e8d-4375-a11d-eec651bb170f/volumes"
Oct 30 21:24:57.997416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:57.993794    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:24:57.997416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:57.993863    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 21:24:58.044230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:58.043771    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 21:24:58.044230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:58.043836    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 21:24:58.432511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:58.430194    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:24:59.008418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:59.004031    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:24:59.008418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:59.004085    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 21:24:59.056414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:59.048008    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 21:24:59.056414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:24:59.048069    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 21:24:59.277113 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:24:59.277018    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:25:00.003800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:00.003761    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:25:00.004375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:00.004343    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 21:25:00.041049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:00.041015    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Oct 30 21:25:00.041202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:00.041066    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Oct 30 21:25:00.759959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:00.759929    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"
Oct 30 21:25:01.010478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:00.993810    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:25:01.010907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:01.010516    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 21:25:01.045531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:01.045500    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:25:01.045994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:01.045982    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:25:01.046654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:01.046637    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:25:01.994784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:01.994748    4918 patch_prober.go:29] interesting pod/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Oct 30 21:25:01.995052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:01.995024    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Oct 30 21:25:03.008414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:03.007774    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:25:03.017648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:03.017618    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:25:04.297983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.297935    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"]
Oct 30 21:25:04.299546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.299518    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5139dc2e-8d18-4d96-b8d4-c91d6d9e6096" containerName="revoke-keys"
Oct 30 21:25:04.299726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.299711    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5139dc2e-8d18-4d96-b8d4-c91d6d9e6096" containerName="revoke-keys"
Oct 30 21:25:04.299853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.299840    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b2f29202-7112-4ba1-a8c0-1c05259e1863" containerName="payload-processing"
Oct 30 21:25:04.299947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.299935    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f29202-7112-4ba1-a8c0-1c05259e1863" containerName="payload-processing"
Oct 30 21:25:04.300049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.300038    4918 cpu_manager.go:414] "containerMap: removing container" podUID="df366a2f-ee2a-44c4-9b91-1d89db6d1475" containerName="payload-processing"
Oct 30 21:25:04.300129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.300118    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df366a2f-ee2a-44c4-9b91-1d89db6d1475" containerName="payload-processing"
Oct 30 21:25:04.300261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.300249    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f8d95733-0e8d-4375-a11d-eec651bb170f" containerName="payload-pre-processing"
Oct 30 21:25:04.300422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.300406    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d95733-0e8d-4375-a11d-eec651bb170f" containerName="payload-pre-processing"
Oct 30 21:25:04.300547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.300532    4918 cpu_manager.go:414] "containerMap: removing container" podUID="056b1a05-afe2-4d7a-ae27-bdac204de67e" containerName="payload-pre-processing"
Oct 30 21:25:04.300636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.300624    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056b1a05-afe2-4d7a-ae27-bdac204de67e" containerName="payload-pre-processing"
Oct 30 21:25:04.300890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.300873    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5139dc2e-8d18-4d96-b8d4-c91d6d9e6096" containerName="revoke-keys"
Oct 30 21:25:04.301018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.301005    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f29202-7112-4ba1-a8c0-1c05259e1863" containerName="payload-processing"
Oct 30 21:25:04.301115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.301104    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="df366a2f-ee2a-44c4-9b91-1d89db6d1475" containerName="payload-processing"
Oct 30 21:25:04.301218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.301206    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d95733-0e8d-4375-a11d-eec651bb170f" containerName="payload-pre-processing"
Oct 30 21:25:04.301313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.301301    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="056b1a05-afe2-4d7a-ae27-bdac204de67e" containerName="payload-pre-processing"
Oct 30 21:25:04.325051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.325023    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"]
Oct 30 21:25:04.325350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.325330    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:04.379300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.376847    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:04.379300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.377605    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:25:04.379300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.378278    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:04.379300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.378348    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:04.467604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.466812    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bcff5c9-jmr8b\" (UID: \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:04.467604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.466869    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pljws\" (UniqueName: \"kubernetes.io/projected/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-kube-api-access-pljws\") pod \"maas-api-revoke-sub-2bcff5c9-jmr8b\" (UID: \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:04.467604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.466929    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bcff5c9-jmr8b\" (UID: \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:04.568111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.568020    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bcff5c9-jmr8b\" (UID: \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:04.568111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.568061    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pljws\" (UniqueName: \"kubernetes.io/projected/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-kube-api-access-pljws\") pod \"maas-api-revoke-sub-2bcff5c9-jmr8b\" (UID: \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:04.568111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.568101    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bcff5c9-jmr8b\" (UID: \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:04.569911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.569878    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bcff5c9-jmr8b\" (UID: \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:04.571083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.571061    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bcff5c9-jmr8b\" (UID: \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:04.577297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.577274    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pljws\" (UniqueName: \"kubernetes.io/projected/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-kube-api-access-pljws\") pod \"maas-api-revoke-sub-2bcff5c9-jmr8b\" (UID: \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:04.777667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:04.777632    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:05.125312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:05.125178    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9_6ab512cc-872e-4d6e-a072-9dec56c6b06e/payload-processing/0.log"
Oct 30 21:25:05.290417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:05.278807    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"]
Oct 30 21:25:05.358786 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:25:05.358737    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66a5aff_0ac0_4d0c_a83e_74f64adbe3ac.slice/crio-b9229d5ce791c4612628ddde61d393b12ef5fe66c8704c7b028439b3ad017104 WatchSource:0}: Error finding container b9229d5ce791c4612628ddde61d393b12ef5fe66c8704c7b028439b3ad017104: Status 404 returned error can't find the container with id b9229d5ce791c4612628ddde61d393b12ef5fe66c8704c7b028439b3ad017104
Oct 30 21:25:05.432458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:05.432423    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b" event={"ID":"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac","Type":"ContainerStarted","Data":"b9229d5ce791c4612628ddde61d393b12ef5fe66c8704c7b028439b3ad017104"}
Oct 30 21:25:06.065098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.065070    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"]
Oct 30 21:25:06.069528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.069494    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:06.077021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.076997    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"]
Oct 30 21:25:06.277774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.277740    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z972r\" (UniqueName: \"kubernetes.io/projected/b38803ee-d4a6-45e3-83b8-762ebefd78a3-kube-api-access-z972r\") pod \"maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6\" (UID: \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:06.277982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.277929    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6\" (UID: \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:06.278332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.278310    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6\" (UID: \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:06.379503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.379383    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6\" (UID: \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:06.379929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.379510    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z972r\" (UniqueName: \"kubernetes.io/projected/b38803ee-d4a6-45e3-83b8-762ebefd78a3-kube-api-access-z972r\") pod \"maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6\" (UID: \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:06.379929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.379605    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6\" (UID: \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:06.380100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.380081    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6\" (UID: \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:06.382023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.382006    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6\" (UID: \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:06.389128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.389104    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z972r\" (UniqueName: \"kubernetes.io/projected/b38803ee-d4a6-45e3-83b8-762ebefd78a3-kube-api-access-z972r\") pod \"maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6\" (UID: \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:06.393721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.393704    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:06.442757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.442722    4918 generic.go:356] "Generic (PLEG): container finished" podID="e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac" containerID="ea08d7260f3293005d811b73031e834614bcbb42eb9665260f6d169f76ae1667" exitCode=0
Oct 30 21:25:06.442934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.442814    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b" event={"ID":"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac","Type":"ContainerDied","Data":"ea08d7260f3293005d811b73031e834614bcbb42eb9665260f6d169f76ae1667"}
Oct 30 21:25:06.679039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:06.679013    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"]
Oct 30 21:25:06.679182 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:25:06.679162    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38803ee_d4a6_45e3_83b8_762ebefd78a3.slice/crio-ea258fd84e03ae0c4b37744a09d2c7a559e6e4821c63d3b88ecffead88a966c5 WatchSource:0}: Error finding container ea258fd84e03ae0c4b37744a09d2c7a559e6e4821c63d3b88ecffead88a966c5: Status 404 returned error can't find the container with id ea258fd84e03ae0c4b37744a09d2c7a559e6e4821c63d3b88ecffead88a966c5
Oct 30 21:25:07.038493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.038466    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"]
Oct 30 21:25:07.042602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.042585    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:07.087450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.086594    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m96bk\" (UniqueName: \"kubernetes.io/projected/cf394595-f240-4c27-bdf6-ea1f94e74617-kube-api-access-m96bk\") pod \"maas-api-revoke-sub-a37c7d39-z5hwj\" (UID: \"cf394595-f240-4c27-bdf6-ea1f94e74617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:07.087450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.086807    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37c7d39-z5hwj\" (UID: \"cf394595-f240-4c27-bdf6-ea1f94e74617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:07.087450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.086863    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37c7d39-z5hwj\" (UID: \"cf394595-f240-4c27-bdf6-ea1f94e74617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:07.090201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.090175    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"]
Oct 30 21:25:07.187994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.187964    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37c7d39-z5hwj\" (UID: \"cf394595-f240-4c27-bdf6-ea1f94e74617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:07.188199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.188029    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37c7d39-z5hwj\" (UID: \"cf394595-f240-4c27-bdf6-ea1f94e74617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:07.188199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.188118    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m96bk\" (UniqueName: \"kubernetes.io/projected/cf394595-f240-4c27-bdf6-ea1f94e74617-kube-api-access-m96bk\") pod \"maas-api-revoke-sub-a37c7d39-z5hwj\" (UID: \"cf394595-f240-4c27-bdf6-ea1f94e74617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:07.188762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.188742    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37c7d39-z5hwj\" (UID: \"cf394595-f240-4c27-bdf6-ea1f94e74617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:07.190762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.190739    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37c7d39-z5hwj\" (UID: \"cf394595-f240-4c27-bdf6-ea1f94e74617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:07.198176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.198147    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m96bk\" (UniqueName: \"kubernetes.io/projected/cf394595-f240-4c27-bdf6-ea1f94e74617-kube-api-access-m96bk\") pod \"maas-api-revoke-sub-a37c7d39-z5hwj\" (UID: \"cf394595-f240-4c27-bdf6-ea1f94e74617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:07.373245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.373166    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:07.456084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.456050    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6" event={"ID":"b38803ee-d4a6-45e3-83b8-762ebefd78a3","Type":"ContainerStarted","Data":"b877e8339a424ad92014e16357645e10492e47b52e3aae37c8e430c8cbd4fdab"}
Oct 30 21:25:07.456537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.456114    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6" event={"ID":"b38803ee-d4a6-45e3-83b8-762ebefd78a3","Type":"ContainerStarted","Data":"ea258fd84e03ae0c4b37744a09d2c7a559e6e4821c63d3b88ecffead88a966c5"}
Oct 30 21:25:07.479783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.479715    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6" podStartSLOduration=1.479694082 podStartE2EDuration="1.479694082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:07.474886026 +0000 UTC m=+1420.521445206" watchObservedRunningTime="2026-10-30 21:25:07.479694082 +0000 UTC m=+1420.526253228"
Oct 30 21:25:07.655162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.655138    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"]
Oct 30 21:25:07.684337 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:25:07.684312    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf394595_f240_4c27_bdf6_ea1f94e74617.slice/crio-f2a7e2c8ff91e27b722ed0d84d74cf4ca28bf756350cfd62fac3cc4b8882a4b2 WatchSource:0}: Error finding container f2a7e2c8ff91e27b722ed0d84d74cf4ca28bf756350cfd62fac3cc4b8882a4b2: Status 404 returned error can't find the container with id f2a7e2c8ff91e27b722ed0d84d74cf4ca28bf756350cfd62fac3cc4b8882a4b2
Oct 30 21:25:07.696342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.696321    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:07.706307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.706286    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-kube-api-access-pljws\" (UniqueName: \"kubernetes.io/projected/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-kube-api-access-pljws\") pod \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\" (UID: \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\") "
Oct 30 21:25:07.706444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.706428    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-xks-ca\") pod \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\" (UID: \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\") "
Oct 30 21:25:07.706558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.706543    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-openshift-ca\") pod \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\" (UID: \"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac\") "
Oct 30 21:25:07.706915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.706890    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-openshift-ca" pod "e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac" (UID: "e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:07.707294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.707266    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:07.708467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.708447    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-kube-api-access-pljws" pod "e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac" (UID: "e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac"). InnerVolumeSpecName "kube-api-access-pljws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:07.708591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.708573    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-xks-ca" pod "e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac" (UID: "e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:07.809113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.809086    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pljws\" (UniqueName: \"kubernetes.io/projected/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-kube-api-access-pljws\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:07.809113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:07.809110    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:08.162110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.162029    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d" podUID="ed90e959-7770-4c1e-82fd-3abfd1800ce2" containerName="main" containerID="cri-o://3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c" gracePeriod=45
Oct 30 21:25:08.413064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.413006    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:25:08.424333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.424299    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tmp-dir\") pod \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") "
Oct 30 21:25:08.424489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.424429    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tls-certs\") pod \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") "
Oct 30 21:25:08.424489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.424466    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-home\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-home\") pod \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") "
Oct 30 21:25:08.424604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.424539    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tmp-dir" pod "ed90e959-7770-4c1e-82fd-3abfd1800ce2" (UID: "ed90e959-7770-4c1e-82fd-3abfd1800ce2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:08.424604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.424562    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-model-cache\") pod \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") "
Oct 30 21:25:08.424604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.424582    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-dshm\") pod \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") "
Oct 30 21:25:08.424772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.424631    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed90e959-7770-4c1e-82fd-3abfd1800ce2-kube-api-access-rb7f4\" (UniqueName: \"kubernetes.io/projected/ed90e959-7770-4c1e-82fd-3abfd1800ce2-kube-api-access-rb7f4\") pod \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\" (UID: \"ed90e959-7770-4c1e-82fd-3abfd1800ce2\") "
Oct 30 21:25:08.424772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.424756    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-model-cache" pod "ed90e959-7770-4c1e-82fd-3abfd1800ce2" (UID: "ed90e959-7770-4c1e-82fd-3abfd1800ce2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:08.424872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.424774    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-home" pod "ed90e959-7770-4c1e-82fd-3abfd1800ce2" (UID: "ed90e959-7770-4c1e-82fd-3abfd1800ce2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:08.425444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.425422    4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:08.425552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.425450    4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:08.425552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.425464    4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:08.426843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.426820    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-dshm" pod "ed90e959-7770-4c1e-82fd-3abfd1800ce2" (UID: "ed90e959-7770-4c1e-82fd-3abfd1800ce2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:08.426956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.426893    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tls-certs" pod "ed90e959-7770-4c1e-82fd-3abfd1800ce2" (UID: "ed90e959-7770-4c1e-82fd-3abfd1800ce2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:08.426956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.426944    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed90e959-7770-4c1e-82fd-3abfd1800ce2-kube-api-access-rb7f4" pod "ed90e959-7770-4c1e-82fd-3abfd1800ce2" (UID: "ed90e959-7770-4c1e-82fd-3abfd1800ce2"). InnerVolumeSpecName "kube-api-access-rb7f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:08.467210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.467162    4918 generic.go:356] "Generic (PLEG): container finished" podID="b38803ee-d4a6-45e3-83b8-762ebefd78a3" containerID="b877e8339a424ad92014e16357645e10492e47b52e3aae37c8e430c8cbd4fdab" exitCode=0
Oct 30 21:25:08.467746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.467721    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6" event={"ID":"b38803ee-d4a6-45e3-83b8-762ebefd78a3","Type":"ContainerDied","Data":"b877e8339a424ad92014e16357645e10492e47b52e3aae37c8e430c8cbd4fdab"}
Oct 30 21:25:08.470154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.470126    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj" event={"ID":"cf394595-f240-4c27-bdf6-ea1f94e74617","Type":"ContainerStarted","Data":"41a0ac9180dd107b9a6d6a3dfa6058b8180055e3ec3551281d5eab0504e13936"}
Oct 30 21:25:08.470272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.470182    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj" event={"ID":"cf394595-f240-4c27-bdf6-ea1f94e74617","Type":"ContainerStarted","Data":"f2a7e2c8ff91e27b722ed0d84d74cf4ca28bf756350cfd62fac3cc4b8882a4b2"}
Oct 30 21:25:08.473532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.473505    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b" event={"ID":"e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac","Type":"ContainerDied","Data":"b9229d5ce791c4612628ddde61d393b12ef5fe66c8704c7b028439b3ad017104"}
Oct 30 21:25:08.473635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.473555    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9229d5ce791c4612628ddde61d393b12ef5fe66c8704c7b028439b3ad017104"
Oct 30 21:25:08.473635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.473603    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"
Oct 30 21:25:08.481492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.481462    4918 generic.go:356] "Generic (PLEG): container finished" podID="ed90e959-7770-4c1e-82fd-3abfd1800ce2" containerID="3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c" exitCode=0
Oct 30 21:25:08.481611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.481495    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d" event={"ID":"ed90e959-7770-4c1e-82fd-3abfd1800ce2","Type":"ContainerDied","Data":"3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c"}
Oct 30 21:25:08.481611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.481530    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d" event={"ID":"ed90e959-7770-4c1e-82fd-3abfd1800ce2","Type":"ContainerDied","Data":"c566715b77bac41674825de176f935e68d5604d2ef5a18bfb87f725556ae2a1c"}
Oct 30 21:25:08.481611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.481553    4918 scope.go:122] "RemoveContainer" containerID="3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c"
Oct 30 21:25:08.482064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.482046    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"
Oct 30 21:25:08.497088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.497059    4918 scope.go:122] "RemoveContainer" containerID="3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c"
Oct 30 21:25:08.497459 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:25:08.497439    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c\": container with ID starting with 3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c not found: ID does not exist" containerID="3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c"
Oct 30 21:25:08.497554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.497469    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c"} err="failed to get container status \"3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c\": rpc error: code = NotFound desc = could not find container \"3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c\": container with ID starting with 3009f2063f58c546570ead3ba53d6a6f21d28f3c6c298cd7f3265689329f908c not found: ID does not exist"
Oct 30 21:25:08.510468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.510421    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj" podStartSLOduration=1.510380499 podStartE2EDuration="1.510380499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:08.509655404 +0000 UTC m=+1421.556214573" watchObservedRunningTime="2026-10-30 21:25:08.510380499 +0000 UTC m=+1421.556939643"
Oct 30 21:25:08.527230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.527201    4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed90e959-7770-4c1e-82fd-3abfd1800ce2-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:08.527230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.527232    4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed90e959-7770-4c1e-82fd-3abfd1800ce2-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:08.527495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.527246    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb7f4\" (UniqueName: \"kubernetes.io/projected/ed90e959-7770-4c1e-82fd-3abfd1800ce2-kube-api-access-rb7f4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:08.527495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.527374    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"]
Oct 30 21:25:08.532529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:08.532506    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-6563b6/praxis-route-6563b6-kserve-78ff97b85c-qt96d"]
Oct 30 21:25:09.077556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.077479    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed90e959-7770-4c1e-82fd-3abfd1800ce2" path="/var/lib/kubelet/pods/ed90e959-7770-4c1e-82fd-3abfd1800ce2/volumes"
Oct 30 21:25:09.495655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.495315    4918 generic.go:356] "Generic (PLEG): container finished" podID="cf394595-f240-4c27-bdf6-ea1f94e74617" containerID="41a0ac9180dd107b9a6d6a3dfa6058b8180055e3ec3551281d5eab0504e13936" exitCode=0
Oct 30 21:25:09.495655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.495439    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj" event={"ID":"cf394595-f240-4c27-bdf6-ea1f94e74617","Type":"ContainerDied","Data":"41a0ac9180dd107b9a6d6a3dfa6058b8180055e3ec3551281d5eab0504e13936"}
Oct 30 21:25:09.736337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.736306    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:09.748952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.748885    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38803ee-d4a6-45e3-83b8-762ebefd78a3-kube-api-access-z972r\" (UniqueName: \"kubernetes.io/projected/b38803ee-d4a6-45e3-83b8-762ebefd78a3-kube-api-access-z972r\") pod \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\" (UID: \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\") "
Oct 30 21:25:09.749060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.748991    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-openshift-ca\") pod \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\" (UID: \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\") "
Oct 30 21:25:09.749060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.749043    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-xks-ca\") pod \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\" (UID: \"b38803ee-d4a6-45e3-83b8-762ebefd78a3\") "
Oct 30 21:25:09.749374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.749351    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-openshift-ca" pod "b38803ee-d4a6-45e3-83b8-762ebefd78a3" (UID: "b38803ee-d4a6-45e3-83b8-762ebefd78a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:09.749612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.749592    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:09.751526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.751503    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-xks-ca" pod "b38803ee-d4a6-45e3-83b8-762ebefd78a3" (UID: "b38803ee-d4a6-45e3-83b8-762ebefd78a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:09.751757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.751728    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38803ee-d4a6-45e3-83b8-762ebefd78a3-kube-api-access-z972r" pod "b38803ee-d4a6-45e3-83b8-762ebefd78a3" (UID: "b38803ee-d4a6-45e3-83b8-762ebefd78a3"). InnerVolumeSpecName "kube-api-access-z972r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:09.850445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.850411    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z972r\" (UniqueName: \"kubernetes.io/projected/b38803ee-d4a6-45e3-83b8-762ebefd78a3-kube-api-access-z972r\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:09.850445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:09.850440    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b38803ee-d4a6-45e3-83b8-762ebefd78a3-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:10.504469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.504441    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"
Oct 30 21:25:10.504897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.504429    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6" event={"ID":"b38803ee-d4a6-45e3-83b8-762ebefd78a3","Type":"ContainerDied","Data":"ea258fd84e03ae0c4b37744a09d2c7a559e6e4821c63d3b88ecffead88a966c5"}
Oct 30 21:25:10.504897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.504517    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea258fd84e03ae0c4b37744a09d2c7a559e6e4821c63d3b88ecffead88a966c5"
Oct 30 21:25:10.654067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.654044    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:10.659622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.659601    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf394595-f240-4c27-bdf6-ea1f94e74617-kube-api-access-m96bk\" (UniqueName: \"kubernetes.io/projected/cf394595-f240-4c27-bdf6-ea1f94e74617-kube-api-access-m96bk\") pod \"cf394595-f240-4c27-bdf6-ea1f94e74617\" (UID: \"cf394595-f240-4c27-bdf6-ea1f94e74617\") "
Oct 30 21:25:10.659749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.659673    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-openshift-ca\") pod \"cf394595-f240-4c27-bdf6-ea1f94e74617\" (UID: \"cf394595-f240-4c27-bdf6-ea1f94e74617\") "
Oct 30 21:25:10.659822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.659779    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-xks-ca\") pod \"cf394595-f240-4c27-bdf6-ea1f94e74617\" (UID: \"cf394595-f240-4c27-bdf6-ea1f94e74617\") "
Oct 30 21:25:10.660008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.659987    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-openshift-ca" pod "cf394595-f240-4c27-bdf6-ea1f94e74617" (UID: "cf394595-f240-4c27-bdf6-ea1f94e74617"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:10.660313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.660294    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:10.661862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.661841    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf394595-f240-4c27-bdf6-ea1f94e74617-kube-api-access-m96bk" pod "cf394595-f240-4c27-bdf6-ea1f94e74617" (UID: "cf394595-f240-4c27-bdf6-ea1f94e74617"). InnerVolumeSpecName "kube-api-access-m96bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:10.661920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.661893    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-xks-ca" pod "cf394595-f240-4c27-bdf6-ea1f94e74617" (UID: "cf394595-f240-4c27-bdf6-ea1f94e74617"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:10.762067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.762000    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf394595-f240-4c27-bdf6-ea1f94e74617-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:10.762067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:10.762027    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m96bk\" (UniqueName: \"kubernetes.io/projected/cf394595-f240-4c27-bdf6-ea1f94e74617-kube-api-access-m96bk\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:11.513977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.513943    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj" event={"ID":"cf394595-f240-4c27-bdf6-ea1f94e74617","Type":"ContainerDied","Data":"f2a7e2c8ff91e27b722ed0d84d74cf4ca28bf756350cfd62fac3cc4b8882a4b2"}
Oct 30 21:25:11.514425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.513994    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2a7e2c8ff91e27b722ed0d84d74cf4ca28bf756350cfd62fac3cc4b8882a4b2"
Oct 30 21:25:11.514425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.513996    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"
Oct 30 21:25:11.532466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.532431    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"]
Oct 30 21:25:11.533067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.533048    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac" containerName="revoke-keys"
Oct 30 21:25:11.533164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.533070    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac" containerName="revoke-keys"
Oct 30 21:25:11.533164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.533090    4918 cpu_manager.go:414] "containerMap: removing container" podUID="cf394595-f240-4c27-bdf6-ea1f94e74617" containerName="revoke-keys"
Oct 30 21:25:11.533164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.533097    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf394595-f240-4c27-bdf6-ea1f94e74617" containerName="revoke-keys"
Oct 30 21:25:11.533164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.533120    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ed90e959-7770-4c1e-82fd-3abfd1800ce2" containerName="main"
Oct 30 21:25:11.533164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.533127    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed90e959-7770-4c1e-82fd-3abfd1800ce2" containerName="main"
Oct 30 21:25:11.533164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.533160    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b38803ee-d4a6-45e3-83b8-762ebefd78a3" containerName="revoke-keys"
Oct 30 21:25:11.533538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.533167    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38803ee-d4a6-45e3-83b8-762ebefd78a3" containerName="revoke-keys"
Oct 30 21:25:11.533538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.533324    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac" containerName="revoke-keys"
Oct 30 21:25:11.533538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.533338    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf394595-f240-4c27-bdf6-ea1f94e74617" containerName="revoke-keys"
Oct 30 21:25:11.533538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.533350    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed90e959-7770-4c1e-82fd-3abfd1800ce2" containerName="main"
Oct 30 21:25:11.533538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.533363    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38803ee-d4a6-45e3-83b8-762ebefd78a3" containerName="revoke-keys"
Oct 30 21:25:11.537076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.537058    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:11.556533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.556509    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:11.556678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.556663    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:11.557223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.557208    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:11.557733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.557705    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:25:11.562027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.562007    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"]
Oct 30 21:25:11.572634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.572611    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph\" (UID: \"a4c5b2fb-356a-4316-8680-67047c28026a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:11.572768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.572746    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph\" (UID: \"a4c5b2fb-356a-4316-8680-67047c28026a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:11.572942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.572922    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j692d\" (UniqueName: \"kubernetes.io/projected/a4c5b2fb-356a-4316-8680-67047c28026a-kube-api-access-j692d\") pod \"maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph\" (UID: \"a4c5b2fb-356a-4316-8680-67047c28026a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:11.674597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.674567    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph\" (UID: \"a4c5b2fb-356a-4316-8680-67047c28026a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:11.674767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.674660    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph\" (UID: \"a4c5b2fb-356a-4316-8680-67047c28026a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:11.674767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.674760    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j692d\" (UniqueName: \"kubernetes.io/projected/a4c5b2fb-356a-4316-8680-67047c28026a-kube-api-access-j692d\") pod \"maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph\" (UID: \"a4c5b2fb-356a-4316-8680-67047c28026a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:11.675175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.675150    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph\" (UID: \"a4c5b2fb-356a-4316-8680-67047c28026a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:11.677172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.677155    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph\" (UID: \"a4c5b2fb-356a-4316-8680-67047c28026a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:11.685456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.685429    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j692d\" (UniqueName: \"kubernetes.io/projected/a4c5b2fb-356a-4316-8680-67047c28026a-kube-api-access-j692d\") pod \"maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph\" (UID: \"a4c5b2fb-356a-4316-8680-67047c28026a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:11.851123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:11.851049    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:12.231999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:12.231969    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"]
Oct 30 21:25:12.232138 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:25:12.232085    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c5b2fb_356a_4316_8680_67047c28026a.slice/crio-f0638f16f0badb78dea300e5f26f8cdd04042a14f195911ef6cec82b37345ce8 WatchSource:0}: Error finding container f0638f16f0badb78dea300e5f26f8cdd04042a14f195911ef6cec82b37345ce8: Status 404 returned error can't find the container with id f0638f16f0badb78dea300e5f26f8cdd04042a14f195911ef6cec82b37345ce8
Oct 30 21:25:12.522667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:12.522637    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph" event={"ID":"a4c5b2fb-356a-4316-8680-67047c28026a","Type":"ContainerStarted","Data":"6044b40e4a15c6750d3370d90be2a2ffd557155fa03a4b32dc01d9f53bb21192"}
Oct 30 21:25:12.522999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:12.522695    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph" event={"ID":"a4c5b2fb-356a-4316-8680-67047c28026a","Type":"ContainerStarted","Data":"f0638f16f0badb78dea300e5f26f8cdd04042a14f195911ef6cec82b37345ce8"}
Oct 30 21:25:12.544448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:12.544405    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph" podStartSLOduration=1.54437634 podStartE2EDuration="1.54437634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:12.54300957 +0000 UTC m=+1425.589568715" watchObservedRunningTime="2026-10-30 21:25:12.54437634 +0000 UTC m=+1425.590935484"
Oct 30 21:25:13.532641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:13.532604    4918 generic.go:356] "Generic (PLEG): container finished" podID="a4c5b2fb-356a-4316-8680-67047c28026a" containerID="6044b40e4a15c6750d3370d90be2a2ffd557155fa03a4b32dc01d9f53bb21192" exitCode=0
Oct 30 21:25:13.533107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:13.532679    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph" event={"ID":"a4c5b2fb-356a-4316-8680-67047c28026a","Type":"ContainerDied","Data":"6044b40e4a15c6750d3370d90be2a2ffd557155fa03a4b32dc01d9f53bb21192"}
Oct 30 21:25:14.775731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:14.775708    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:14.827321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:14.827243    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-xks-ca\") pod \"a4c5b2fb-356a-4316-8680-67047c28026a\" (UID: \"a4c5b2fb-356a-4316-8680-67047c28026a\") "
Oct 30 21:25:14.827321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:14.827279    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-openshift-ca\") pod \"a4c5b2fb-356a-4316-8680-67047c28026a\" (UID: \"a4c5b2fb-356a-4316-8680-67047c28026a\") "
Oct 30 21:25:14.827518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:14.827345    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c5b2fb-356a-4316-8680-67047c28026a-kube-api-access-j692d\" (UniqueName: \"kubernetes.io/projected/a4c5b2fb-356a-4316-8680-67047c28026a-kube-api-access-j692d\") pod \"a4c5b2fb-356a-4316-8680-67047c28026a\" (UID: \"a4c5b2fb-356a-4316-8680-67047c28026a\") "
Oct 30 21:25:14.827777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:14.827736    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-openshift-ca" pod "a4c5b2fb-356a-4316-8680-67047c28026a" (UID: "a4c5b2fb-356a-4316-8680-67047c28026a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:14.827986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:14.827964    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:14.829792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:14.829764    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-xks-ca" pod "a4c5b2fb-356a-4316-8680-67047c28026a" (UID: "a4c5b2fb-356a-4316-8680-67047c28026a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:14.829940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:14.829918    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c5b2fb-356a-4316-8680-67047c28026a-kube-api-access-j692d" pod "a4c5b2fb-356a-4316-8680-67047c28026a" (UID: "a4c5b2fb-356a-4316-8680-67047c28026a"). InnerVolumeSpecName "kube-api-access-j692d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:14.929050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:14.929022    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c5b2fb-356a-4316-8680-67047c28026a-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:14.929050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:14.929048    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j692d\" (UniqueName: \"kubernetes.io/projected/a4c5b2fb-356a-4316-8680-67047c28026a-kube-api-access-j692d\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:15.579566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:15.578067    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph" event={"ID":"a4c5b2fb-356a-4316-8680-67047c28026a","Type":"ContainerDied","Data":"f0638f16f0badb78dea300e5f26f8cdd04042a14f195911ef6cec82b37345ce8"}
Oct 30 21:25:15.579566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:15.578133    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0638f16f0badb78dea300e5f26f8cdd04042a14f195911ef6cec82b37345ce8"
Oct 30 21:25:15.579566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:15.578198    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"
Oct 30 21:25:17.339417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.338442    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"]
Oct 30 21:25:17.339417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.339326    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a4c5b2fb-356a-4316-8680-67047c28026a" containerName="revoke-keys"
Oct 30 21:25:17.339417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.339346    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c5b2fb-356a-4316-8680-67047c28026a" containerName="revoke-keys"
Oct 30 21:25:17.340080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.339556    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c5b2fb-356a-4316-8680-67047c28026a" containerName="revoke-keys"
Oct 30 21:25:17.550149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.545711    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:17.554433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.552662    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:17.554433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.553322    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:17.554433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.553444    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:25:17.554433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.553514    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:17.584416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.584117    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"]
Oct 30 21:25:17.679768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.679729    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e10b7-cwq86\" (UID: \"f36bc838-0a59-4ed5-af45-0db8bb269691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:17.680130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.680108    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e10b7-cwq86\" (UID: \"f36bc838-0a59-4ed5-af45-0db8bb269691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:17.680482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.680452    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxzm\" (UniqueName: \"kubernetes.io/projected/f36bc838-0a59-4ed5-af45-0db8bb269691-kube-api-access-lfxzm\") pod \"maas-api-revoke-sub-ef2e10b7-cwq86\" (UID: \"f36bc838-0a59-4ed5-af45-0db8bb269691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:17.784413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.783872    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e10b7-cwq86\" (UID: \"f36bc838-0a59-4ed5-af45-0db8bb269691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:17.784413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.784029    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e10b7-cwq86\" (UID: \"f36bc838-0a59-4ed5-af45-0db8bb269691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:17.784413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.784164    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxzm\" (UniqueName: \"kubernetes.io/projected/f36bc838-0a59-4ed5-af45-0db8bb269691-kube-api-access-lfxzm\") pod \"maas-api-revoke-sub-ef2e10b7-cwq86\" (UID: \"f36bc838-0a59-4ed5-af45-0db8bb269691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:17.793417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.785374    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e10b7-cwq86\" (UID: \"f36bc838-0a59-4ed5-af45-0db8bb269691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:17.809101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.809069    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxzm\" (UniqueName: \"kubernetes.io/projected/f36bc838-0a59-4ed5-af45-0db8bb269691-kube-api-access-lfxzm\") pod \"maas-api-revoke-sub-ef2e10b7-cwq86\" (UID: \"f36bc838-0a59-4ed5-af45-0db8bb269691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:17.809751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.809700    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e10b7-cwq86\" (UID: \"f36bc838-0a59-4ed5-af45-0db8bb269691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:17.984652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:17.984624    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:18.327481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:18.326071    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"]
Oct 30 21:25:18.729171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:18.729139    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86" event={"ID":"f36bc838-0a59-4ed5-af45-0db8bb269691","Type":"ContainerStarted","Data":"c6c7a9aba95675f177fd888f3456665b7321b891973e34ff6fabfa388f936edb"}
Oct 30 21:25:18.729514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:18.729200    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86" event={"ID":"f36bc838-0a59-4ed5-af45-0db8bb269691","Type":"ContainerStarted","Data":"a0af34b3925029039ab91e351ddc0766e99e71e4fd4bcca1eea739a12f331a16"}
Oct 30 21:25:18.749903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:18.749853    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86" podStartSLOduration=1.7498384599999999 podStartE2EDuration="1.74983846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:18.746456651 +0000 UTC m=+1431.793015820" watchObservedRunningTime="2026-10-30 21:25:18.74983846 +0000 UTC m=+1431.796397596"
Oct 30 21:25:19.758449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:19.757593    4918 generic.go:356] "Generic (PLEG): container finished" podID="f36bc838-0a59-4ed5-af45-0db8bb269691" containerID="c6c7a9aba95675f177fd888f3456665b7321b891973e34ff6fabfa388f936edb" exitCode=0
Oct 30 21:25:19.758449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:19.757667    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86" event={"ID":"f36bc838-0a59-4ed5-af45-0db8bb269691","Type":"ContainerDied","Data":"c6c7a9aba95675f177fd888f3456665b7321b891973e34ff6fabfa388f936edb"}
Oct 30 21:25:20.956857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:20.956825    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:21.083407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:21.083310    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-openshift-ca\") pod \"f36bc838-0a59-4ed5-af45-0db8bb269691\" (UID: \"f36bc838-0a59-4ed5-af45-0db8bb269691\") "
Oct 30 21:25:21.083562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:21.083538    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36bc838-0a59-4ed5-af45-0db8bb269691-kube-api-access-lfxzm\" (UniqueName: \"kubernetes.io/projected/f36bc838-0a59-4ed5-af45-0db8bb269691-kube-api-access-lfxzm\") pod \"f36bc838-0a59-4ed5-af45-0db8bb269691\" (UID: \"f36bc838-0a59-4ed5-af45-0db8bb269691\") "
Oct 30 21:25:21.083655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:21.083634    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-openshift-ca" pod "f36bc838-0a59-4ed5-af45-0db8bb269691" (UID: "f36bc838-0a59-4ed5-af45-0db8bb269691"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:21.083705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:21.083650    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-xks-ca\") pod \"f36bc838-0a59-4ed5-af45-0db8bb269691\" (UID: \"f36bc838-0a59-4ed5-af45-0db8bb269691\") "
Oct 30 21:25:21.084264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:21.084237    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:21.115461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:21.113510    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-xks-ca" pod "f36bc838-0a59-4ed5-af45-0db8bb269691" (UID: "f36bc838-0a59-4ed5-af45-0db8bb269691"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:21.115461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:21.114562    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36bc838-0a59-4ed5-af45-0db8bb269691-kube-api-access-lfxzm" pod "f36bc838-0a59-4ed5-af45-0db8bb269691" (UID: "f36bc838-0a59-4ed5-af45-0db8bb269691"). InnerVolumeSpecName "kube-api-access-lfxzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:21.243264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:21.211711    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfxzm\" (UniqueName: \"kubernetes.io/projected/f36bc838-0a59-4ed5-af45-0db8bb269691-kube-api-access-lfxzm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:21.243264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:21.211748    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36bc838-0a59-4ed5-af45-0db8bb269691-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:21.778945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:21.778907    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86" event={"ID":"f36bc838-0a59-4ed5-af45-0db8bb269691","Type":"ContainerDied","Data":"a0af34b3925029039ab91e351ddc0766e99e71e4fd4bcca1eea739a12f331a16"}
Oct 30 21:25:21.779149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:21.778961    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0af34b3925029039ab91e351ddc0766e99e71e4fd4bcca1eea739a12f331a16"
Oct 30 21:25:21.779149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:21.778918    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"
Oct 30 21:25:23.469704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:23.469667    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"]
Oct 30 21:25:23.470883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:23.470848    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c" podUID="1103cf6f-4d84-4537-8da8-84fcbee8b32d" containerName="payload-processing" containerID="cri-o://50540d3446351e2b8b655064076ae872e8cad1407ab64c7f29d6245cc3f0b4e2" gracePeriod=30
Oct 30 21:25:23.924421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:23.924351    4918 generic.go:356] "Generic (PLEG): container finished" podID="1103cf6f-4d84-4537-8da8-84fcbee8b32d" containerID="50540d3446351e2b8b655064076ae872e8cad1407ab64c7f29d6245cc3f0b4e2" exitCode=0
Oct 30 21:25:23.924953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:23.924921    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c" event={"ID":"1103cf6f-4d84-4537-8da8-84fcbee8b32d","Type":"ContainerDied","Data":"50540d3446351e2b8b655064076ae872e8cad1407ab64c7f29d6245cc3f0b4e2"}
Oct 30 21:25:23.925270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:23.925250    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"]
Oct 30 21:25:23.932947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:23.932918    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0fc028-z8tf6"]
Oct 30 21:25:23.985852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:23.985824    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"
Oct 30 21:25:24.176743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:24.176662    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1103cf6f-4d84-4537-8da8-84fcbee8b32d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1103cf6f-4d84-4537-8da8-84fcbee8b32d-plugins-config-volume\") pod \"1103cf6f-4d84-4537-8da8-84fcbee8b32d\" (UID: \"1103cf6f-4d84-4537-8da8-84fcbee8b32d\") "
Oct 30 21:25:24.176923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:24.176825    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1103cf6f-4d84-4537-8da8-84fcbee8b32d-kube-api-access-qfxs7\" (UniqueName: \"kubernetes.io/projected/1103cf6f-4d84-4537-8da8-84fcbee8b32d-kube-api-access-qfxs7\") pod \"1103cf6f-4d84-4537-8da8-84fcbee8b32d\" (UID: \"1103cf6f-4d84-4537-8da8-84fcbee8b32d\") "
Oct 30 21:25:24.177082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:24.177053    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1103cf6f-4d84-4537-8da8-84fcbee8b32d-plugins-config-volume" pod "1103cf6f-4d84-4537-8da8-84fcbee8b32d" (UID: "1103cf6f-4d84-4537-8da8-84fcbee8b32d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:24.177527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:24.177500    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1103cf6f-4d84-4537-8da8-84fcbee8b32d-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:24.179497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:24.179429    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1103cf6f-4d84-4537-8da8-84fcbee8b32d-kube-api-access-qfxs7" pod "1103cf6f-4d84-4537-8da8-84fcbee8b32d" (UID: "1103cf6f-4d84-4537-8da8-84fcbee8b32d"). InnerVolumeSpecName "kube-api-access-qfxs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:24.278952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:24.278916    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfxs7\" (UniqueName: \"kubernetes.io/projected/1103cf6f-4d84-4537-8da8-84fcbee8b32d-kube-api-access-qfxs7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:24.977509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:24.970918    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c" event={"ID":"1103cf6f-4d84-4537-8da8-84fcbee8b32d","Type":"ContainerDied","Data":"7fa778a9fc581fdbb2177800bf1d79134fa953762ae55ff01b790abf7f91d99b"}
Oct 30 21:25:24.977509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:24.971047    4918 scope.go:122] "RemoveContainer" containerID="50540d3446351e2b8b655064076ae872e8cad1407ab64c7f29d6245cc3f0b4e2"
Oct 30 21:25:24.977509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:24.971338    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"
Oct 30 21:25:25.022659 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:25:25.022629    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1103cf6f_4d84_4537_8da8_84fcbee8b32d.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:25:25.022830 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:25:25.022635    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1103cf6f_4d84_4537_8da8_84fcbee8b32d.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 21:25:25.033485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:25.033455    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"]
Oct 30 21:25:25.037130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:25.037109    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0fc028-cc589f95b-qgg9c"]
Oct 30 21:25:25.078013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:25.077982    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1103cf6f-4d84-4537-8da8-84fcbee8b32d" path="/var/lib/kubelet/pods/1103cf6f-4d84-4537-8da8-84fcbee8b32d/volumes"
Oct 30 21:25:25.078514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:25.078500    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b38803ee-d4a6-45e3-83b8-762ebefd78a3" path="/var/lib/kubelet/pods/b38803ee-d4a6-45e3-83b8-762ebefd78a3/volumes"
Oct 30 21:25:27.862738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:27.862706    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"]
Oct 30 21:25:27.865975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:27.865953    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7db332-jgtph"]
Oct 30 21:25:27.999088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:27.999053    4918 generic.go:356] "Generic (PLEG): container finished" podID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerID="2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223" exitCode=0
Oct 30 21:25:27.999247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:27.999128    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" event={"ID":"a63cdd9a-24cd-4674-b165-a21d9781f974","Type":"ContainerDied","Data":"2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223"}
Oct 30 21:25:27.999595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:27.999578    4918 scope.go:122] "RemoveContainer" containerID="2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223"
Oct 30 21:25:28.001118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:28.001085    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:25:28.001209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:28.001153    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:25:28.001209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:28.001187    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:25:29.019269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:29.019235    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" event={"ID":"a63cdd9a-24cd-4674-b165-a21d9781f974","Type":"ContainerStarted","Data":"81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4"}
Oct 30 21:25:29.077942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:29.077915    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c5b2fb-356a-4316-8680-67047c28026a" path="/var/lib/kubelet/pods/a4c5b2fb-356a-4316-8680-67047c28026a/volumes"
Oct 30 21:25:29.160673 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:25:29.160631    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:25:29.420323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:29.420240    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:25:29.421783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:29.421758    4918 patch_prober.go:29] interesting pod/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:25:29.421907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:29.421810    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 21:25:30.218419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:30.213167    4918 scope.go:122] "RemoveContainer" containerID="f59673a9befabe5b24a47c03cc7ac80b34a54ae303614ec3dedc225007142887"
Oct 30 21:25:30.234948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:30.234928    4918 scope.go:122] "RemoveContainer" containerID="ac77274f2431d7200d45d1d4aa2dbe51e5ab190c68f0f5e5e547195d140977df"
Oct 30 21:25:30.244423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:30.244370    4918 scope.go:122] "RemoveContainer" containerID="bd0843013f28fd895a5c7d815481a71018c687cc973dc912a0130885c22954b0"
Oct 30 21:25:30.254311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:30.254285    4918 scope.go:122] "RemoveContainer" containerID="4440e7edd25f05580a9912e8cfb892d1f6c0fb9ab669118b9847f739992363ce"
Oct 30 21:25:30.264484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:30.264465    4918 scope.go:122] "RemoveContainer" containerID="f091cb6ff01501804755445088bdbc7bd65a62323a75818399e92384b7d2cc5b"
Oct 30 21:25:30.420091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:30.420057    4918 patch_prober.go:29] interesting pod/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:25:30.420255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:30.420126    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 21:25:31.433414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:31.426724    4918 patch_prober.go:29] interesting pod/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:25:31.433414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:31.426834    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 21:25:32.420310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.420273    4918 patch_prober.go:29] interesting pod/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:25:32.420539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.420333    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 21:25:32.644520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.644485    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"]
Oct 30 21:25:32.645055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.645041    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f36bc838-0a59-4ed5-af45-0db8bb269691" containerName="revoke-keys"
Oct 30 21:25:32.645097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.645057    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36bc838-0a59-4ed5-af45-0db8bb269691" containerName="revoke-keys"
Oct 30 21:25:32.645133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.645101    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1103cf6f-4d84-4537-8da8-84fcbee8b32d" containerName="payload-processing"
Oct 30 21:25:32.645133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.645106    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1103cf6f-4d84-4537-8da8-84fcbee8b32d" containerName="payload-processing"
Oct 30 21:25:32.645242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.645187    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1103cf6f-4d84-4537-8da8-84fcbee8b32d" containerName="payload-processing"
Oct 30 21:25:32.645242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.645199    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36bc838-0a59-4ed5-af45-0db8bb269691" containerName="revoke-keys"
Oct 30 21:25:32.650688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.650666    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:32.657898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.657764    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:32.657898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.657858    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:32.657898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.657868    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:25:32.658154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.657912    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:32.658489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.658469    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"]
Oct 30 21:25:32.763423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.759807    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e5d7d0-nwmv6\" (UID: \"c77f820a-500a-4201-9fbd-1d470b63971b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:32.763423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.759917    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4bqn\" (UniqueName: \"kubernetes.io/projected/c77f820a-500a-4201-9fbd-1d470b63971b-kube-api-access-r4bqn\") pod \"maas-api-revoke-sub-87e5d7d0-nwmv6\" (UID: \"c77f820a-500a-4201-9fbd-1d470b63971b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:32.763423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.760011    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e5d7d0-nwmv6\" (UID: \"c77f820a-500a-4201-9fbd-1d470b63971b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:32.870414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.861343    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e5d7d0-nwmv6\" (UID: \"c77f820a-500a-4201-9fbd-1d470b63971b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:32.870414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.861499    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4bqn\" (UniqueName: \"kubernetes.io/projected/c77f820a-500a-4201-9fbd-1d470b63971b-kube-api-access-r4bqn\") pod \"maas-api-revoke-sub-87e5d7d0-nwmv6\" (UID: \"c77f820a-500a-4201-9fbd-1d470b63971b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:32.870414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.861619    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e5d7d0-nwmv6\" (UID: \"c77f820a-500a-4201-9fbd-1d470b63971b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:32.870414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.862719    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e5d7d0-nwmv6\" (UID: \"c77f820a-500a-4201-9fbd-1d470b63971b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:32.895417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.882464    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4bqn\" (UniqueName: \"kubernetes.io/projected/c77f820a-500a-4201-9fbd-1d470b63971b-kube-api-access-r4bqn\") pod \"maas-api-revoke-sub-87e5d7d0-nwmv6\" (UID: \"c77f820a-500a-4201-9fbd-1d470b63971b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:32.931437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.922597    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e5d7d0-nwmv6\" (UID: \"c77f820a-500a-4201-9fbd-1d470b63971b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:32.970076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:32.969837    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:33.224858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.224830    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"]
Oct 30 21:25:33.230200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.230177    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:33.249169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.249144    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"]
Oct 30 21:25:33.276116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.276075    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc7nc\" (UniqueName: \"kubernetes.io/projected/c75cee27-f8d0-4491-b218-2bda586ed9f8-kube-api-access-cc7nc\") pod \"maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn\" (UID: \"c75cee27-f8d0-4491-b218-2bda586ed9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:33.276468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.276318    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn\" (UID: \"c75cee27-f8d0-4491-b218-2bda586ed9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:33.276557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.276488    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn\" (UID: \"c75cee27-f8d0-4491-b218-2bda586ed9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:33.314451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.314380    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"]
Oct 30 21:25:33.377776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.377738    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn\" (UID: \"c75cee27-f8d0-4491-b218-2bda586ed9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:33.377910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.377828    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn\" (UID: \"c75cee27-f8d0-4491-b218-2bda586ed9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:33.377991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.377980    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc7nc\" (UniqueName: \"kubernetes.io/projected/c75cee27-f8d0-4491-b218-2bda586ed9f8-kube-api-access-cc7nc\") pod \"maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn\" (UID: \"c75cee27-f8d0-4491-b218-2bda586ed9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:33.378541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.378438    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn\" (UID: \"c75cee27-f8d0-4491-b218-2bda586ed9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:33.380251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.380229    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn\" (UID: \"c75cee27-f8d0-4491-b218-2bda586ed9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:33.420587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.420551    4918 patch_prober.go:29] interesting pod/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Oct 30 21:25:33.420758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.420604    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Oct 30 21:25:33.505415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.500579    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc7nc\" (UniqueName: \"kubernetes.io/projected/c75cee27-f8d0-4491-b218-2bda586ed9f8-kube-api-access-cc7nc\") pod \"maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn\" (UID: \"c75cee27-f8d0-4491-b218-2bda586ed9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:33.544882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.544855    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:33.721920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.721894    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:25:33.850740 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:25:33.850711    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75cee27_f8d0_4491_b218_2bda586ed9f8.slice/crio-4f9d8ab3bb0cc36fc2f3f2f647a6005dd3540b4c3e8316d72d8426857bfb35c5 WatchSource:0}: Error finding container 4f9d8ab3bb0cc36fc2f3f2f647a6005dd3540b4c3e8316d72d8426857bfb35c5: Status 404 returned error can't find the container with id 4f9d8ab3bb0cc36fc2f3f2f647a6005dd3540b4c3e8316d72d8426857bfb35c5
Oct 30 21:25:33.851251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:33.851225    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"]
Oct 30 21:25:34.260886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:34.260851    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6" event={"ID":"c77f820a-500a-4201-9fbd-1d470b63971b","Type":"ContainerStarted","Data":"8051a32189a028a3ee0b50a1fab9f61db1b9a2a2e6cf581830a59a60d906fb35"}
Oct 30 21:25:34.261108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:34.260917    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6" event={"ID":"c77f820a-500a-4201-9fbd-1d470b63971b","Type":"ContainerStarted","Data":"79b6db3b0853fd26f37eb6bacb67d3c9ed47632a034d74a0ff6eede698c2d89a"}
Oct 30 21:25:34.262576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:34.262552    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn" event={"ID":"c75cee27-f8d0-4491-b218-2bda586ed9f8","Type":"ContainerStarted","Data":"4f9d8ab3bb0cc36fc2f3f2f647a6005dd3540b4c3e8316d72d8426857bfb35c5"}
Oct 30 21:25:34.281020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:34.280975    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6" podStartSLOduration=2.280963049 podStartE2EDuration="2.280963049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:34.280245996 +0000 UTC m=+1447.326805160" watchObservedRunningTime="2026-10-30 21:25:34.280963049 +0000 UTC m=+1447.327522193"
Oct 30 21:25:34.443440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:34.443371    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:25:34.444878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:34.444858    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:25:35.271708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:35.271677    4918 generic.go:356] "Generic (PLEG): container finished" podID="c75cee27-f8d0-4491-b218-2bda586ed9f8" containerID="4014a0830e047858301ca5ffeb6472de7fdd65f6f9f82e90af5b4abc8729b577" exitCode=0
Oct 30 21:25:35.272176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:35.271772    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn" event={"ID":"c75cee27-f8d0-4491-b218-2bda586ed9f8","Type":"ContainerDied","Data":"4014a0830e047858301ca5ffeb6472de7fdd65f6f9f82e90af5b4abc8729b577"}
Oct 30 21:25:35.273556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:35.273535    4918 generic.go:356] "Generic (PLEG): container finished" podID="c77f820a-500a-4201-9fbd-1d470b63971b" containerID="8051a32189a028a3ee0b50a1fab9f61db1b9a2a2e6cf581830a59a60d906fb35" exitCode=0
Oct 30 21:25:35.273676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:35.273591    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6" event={"ID":"c77f820a-500a-4201-9fbd-1d470b63971b","Type":"ContainerDied","Data":"8051a32189a028a3ee0b50a1fab9f61db1b9a2a2e6cf581830a59a60d906fb35"}
Oct 30 21:25:35.715823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:35.715791    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"]
Oct 30 21:25:35.716067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:35.716041    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy" containerID="cri-o://119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7" gracePeriod=30
Oct 30 21:25:36.201735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.201702    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"]
Oct 30 21:25:36.206670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.206531    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:36.212965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.212938    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"]
Oct 30 21:25:36.373017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.372631    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22050016-659c-4232-be33-74bcc81d8535-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9\" (UID: \"22050016-659c-4232-be33-74bcc81d8535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:36.373017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.372773    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22050016-659c-4232-be33-74bcc81d8535-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9\" (UID: \"22050016-659c-4232-be33-74bcc81d8535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:36.373017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.372834    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m947p\" (UniqueName: \"kubernetes.io/projected/22050016-659c-4232-be33-74bcc81d8535-kube-api-access-m947p\") pod \"maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9\" (UID: \"22050016-659c-4232-be33-74bcc81d8535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:36.443839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.443810    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:36.477849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.476049    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m947p\" (UniqueName: \"kubernetes.io/projected/22050016-659c-4232-be33-74bcc81d8535-kube-api-access-m947p\") pod \"maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9\" (UID: \"22050016-659c-4232-be33-74bcc81d8535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:36.477849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.476240    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22050016-659c-4232-be33-74bcc81d8535-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9\" (UID: \"22050016-659c-4232-be33-74bcc81d8535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:36.477849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.476459    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22050016-659c-4232-be33-74bcc81d8535-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9\" (UID: \"22050016-659c-4232-be33-74bcc81d8535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:36.477849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.477436    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22050016-659c-4232-be33-74bcc81d8535-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9\" (UID: \"22050016-659c-4232-be33-74bcc81d8535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:36.491970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.491936    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m947p\" (UniqueName: \"kubernetes.io/projected/22050016-659c-4232-be33-74bcc81d8535-kube-api-access-m947p\") pod \"maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9\" (UID: \"22050016-659c-4232-be33-74bcc81d8535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:36.492743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.492721    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22050016-659c-4232-be33-74bcc81d8535-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9\" (UID: \"22050016-659c-4232-be33-74bcc81d8535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:36.509972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.509951    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:36.530636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.530607    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:36.578336 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.577540    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c77f820a-500a-4201-9fbd-1d470b63971b-kube-api-access-r4bqn\" (UniqueName: \"kubernetes.io/projected/c77f820a-500a-4201-9fbd-1d470b63971b-kube-api-access-r4bqn\") pod \"c77f820a-500a-4201-9fbd-1d470b63971b\" (UID: \"c77f820a-500a-4201-9fbd-1d470b63971b\") "
Oct 30 21:25:36.578336 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.577728    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-xks-ca\") pod \"c77f820a-500a-4201-9fbd-1d470b63971b\" (UID: \"c77f820a-500a-4201-9fbd-1d470b63971b\") "
Oct 30 21:25:36.578336 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.577831    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-openshift-ca\") pod \"c77f820a-500a-4201-9fbd-1d470b63971b\" (UID: \"c77f820a-500a-4201-9fbd-1d470b63971b\") "
Oct 30 21:25:36.580579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.580470    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-openshift-ca" pod "c77f820a-500a-4201-9fbd-1d470b63971b" (UID: "c77f820a-500a-4201-9fbd-1d470b63971b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:36.580712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.580566    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77f820a-500a-4201-9fbd-1d470b63971b-kube-api-access-r4bqn" pod "c77f820a-500a-4201-9fbd-1d470b63971b" (UID: "c77f820a-500a-4201-9fbd-1d470b63971b"). InnerVolumeSpecName "kube-api-access-r4bqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:36.581342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.581320    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-xks-ca" pod "c77f820a-500a-4201-9fbd-1d470b63971b" (UID: "c77f820a-500a-4201-9fbd-1d470b63971b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:36.679790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.679761    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-xks-ca\") pod \"c75cee27-f8d0-4491-b218-2bda586ed9f8\" (UID: \"c75cee27-f8d0-4491-b218-2bda586ed9f8\") "
Oct 30 21:25:36.680001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.679842    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75cee27-f8d0-4491-b218-2bda586ed9f8-kube-api-access-cc7nc\" (UniqueName: \"kubernetes.io/projected/c75cee27-f8d0-4491-b218-2bda586ed9f8-kube-api-access-cc7nc\") pod \"c75cee27-f8d0-4491-b218-2bda586ed9f8\" (UID: \"c75cee27-f8d0-4491-b218-2bda586ed9f8\") "
Oct 30 21:25:36.680001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.679902    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-openshift-ca\") pod \"c75cee27-f8d0-4491-b218-2bda586ed9f8\" (UID: \"c75cee27-f8d0-4491-b218-2bda586ed9f8\") "
Oct 30 21:25:36.680295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.680269    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-openshift-ca" pod "c75cee27-f8d0-4491-b218-2bda586ed9f8" (UID: "c75cee27-f8d0-4491-b218-2bda586ed9f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:36.680672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.680650    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:36.680755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.680676    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:36.680755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.680689    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4bqn\" (UniqueName: \"kubernetes.io/projected/c77f820a-500a-4201-9fbd-1d470b63971b-kube-api-access-r4bqn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:36.680755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.680703    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c77f820a-500a-4201-9fbd-1d470b63971b-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:36.682134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.682109    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75cee27-f8d0-4491-b218-2bda586ed9f8-kube-api-access-cc7nc" pod "c75cee27-f8d0-4491-b218-2bda586ed9f8" (UID: "c75cee27-f8d0-4491-b218-2bda586ed9f8"). InnerVolumeSpecName "kube-api-access-cc7nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:36.682263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.682133    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-xks-ca" pod "c75cee27-f8d0-4491-b218-2bda586ed9f8" (UID: "c75cee27-f8d0-4491-b218-2bda586ed9f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:36.787270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.787238    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75cee27-f8d0-4491-b218-2bda586ed9f8-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:36.787566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.787514    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc7nc\" (UniqueName: \"kubernetes.io/projected/c75cee27-f8d0-4491-b218-2bda586ed9f8-kube-api-access-cc7nc\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:36.843046 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:25:36.843014    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22050016_659c_4232_be33_74bcc81d8535.slice/crio-d4d067ef2629949703ecdcd892cae07e9118f26402b72ad53a76368ab48affa9 WatchSource:0}: Error finding container d4d067ef2629949703ecdcd892cae07e9118f26402b72ad53a76368ab48affa9: Status 404 returned error can't find the container with id d4d067ef2629949703ecdcd892cae07e9118f26402b72ad53a76368ab48affa9
Oct 30 21:25:36.843858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:36.843832    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"]
Oct 30 21:25:37.289277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:37.289247    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:25:37.289487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:37.289283    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6" event={"ID":"c77f820a-500a-4201-9fbd-1d470b63971b","Type":"ContainerDied","Data":"79b6db3b0853fd26f37eb6bacb67d3c9ed47632a034d74a0ff6eede698c2d89a"}
Oct 30 21:25:37.289487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:37.289336    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79b6db3b0853fd26f37eb6bacb67d3c9ed47632a034d74a0ff6eede698c2d89a"
Oct 30 21:25:37.290878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:37.290860    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"
Oct 30 21:25:37.291017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:37.290876    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn" event={"ID":"c75cee27-f8d0-4491-b218-2bda586ed9f8","Type":"ContainerDied","Data":"4f9d8ab3bb0cc36fc2f3f2f647a6005dd3540b4c3e8316d72d8426857bfb35c5"}
Oct 30 21:25:37.291017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:37.290917    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f9d8ab3bb0cc36fc2f3f2f647a6005dd3540b4c3e8316d72d8426857bfb35c5"
Oct 30 21:25:37.293300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:37.293277    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9" event={"ID":"22050016-659c-4232-be33-74bcc81d8535","Type":"ContainerStarted","Data":"a985bc1b4882ea00ae3dccc336a0c9963732489b73f93ea751ee23ffc8331641"}
Oct 30 21:25:37.293417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:37.293324    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9" event={"ID":"22050016-659c-4232-be33-74bcc81d8535","Type":"ContainerStarted","Data":"d4d067ef2629949703ecdcd892cae07e9118f26402b72ad53a76368ab48affa9"}
Oct 30 21:25:37.336553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:37.336501    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9" podStartSLOduration=1.336488926 podStartE2EDuration="1.336488926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:37.332688109 +0000 UTC m=+1450.379247256" watchObservedRunningTime="2026-10-30 21:25:37.336488926 +0000 UTC m=+1450.383048071"
Oct 30 21:25:38.311445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:38.310849    4918 generic.go:356] "Generic (PLEG): container finished" podID="22050016-659c-4232-be33-74bcc81d8535" containerID="a985bc1b4882ea00ae3dccc336a0c9963732489b73f93ea751ee23ffc8331641" exitCode=0
Oct 30 21:25:38.311445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:38.310966    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9" event={"ID":"22050016-659c-4232-be33-74bcc81d8535","Type":"ContainerDied","Data":"a985bc1b4882ea00ae3dccc336a0c9963732489b73f93ea751ee23ffc8331641"}
Oct 30 21:25:38.910418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:38.898787    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"]
Oct 30 21:25:38.927415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:38.914601    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" podUID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" containerName="istio-proxy" containerID="cri-o://3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910" gracePeriod=30
Oct 30 21:25:39.838166 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:39.838137    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:39.897048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:39.897013    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22050016-659c-4232-be33-74bcc81d8535-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22050016-659c-4232-be33-74bcc81d8535-maas-api-xks-ca\") pod \"22050016-659c-4232-be33-74bcc81d8535\" (UID: \"22050016-659c-4232-be33-74bcc81d8535\") "
Oct 30 21:25:39.897933 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:39.897909    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22050016-659c-4232-be33-74bcc81d8535-kube-api-access-m947p\" (UniqueName: \"kubernetes.io/projected/22050016-659c-4232-be33-74bcc81d8535-kube-api-access-m947p\") pod \"22050016-659c-4232-be33-74bcc81d8535\" (UID: \"22050016-659c-4232-be33-74bcc81d8535\") "
Oct 30 21:25:39.898549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:39.898529    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22050016-659c-4232-be33-74bcc81d8535-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22050016-659c-4232-be33-74bcc81d8535-maas-api-openshift-ca\") pod \"22050016-659c-4232-be33-74bcc81d8535\" (UID: \"22050016-659c-4232-be33-74bcc81d8535\") "
Oct 30 21:25:39.899984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:39.899354    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22050016-659c-4232-be33-74bcc81d8535-maas-api-openshift-ca" pod "22050016-659c-4232-be33-74bcc81d8535" (UID: "22050016-659c-4232-be33-74bcc81d8535"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:39.901249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:39.901226    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22050016-659c-4232-be33-74bcc81d8535-kube-api-access-m947p" pod "22050016-659c-4232-be33-74bcc81d8535" (UID: "22050016-659c-4232-be33-74bcc81d8535"). InnerVolumeSpecName "kube-api-access-m947p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:39.919503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:39.918901    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22050016-659c-4232-be33-74bcc81d8535-maas-api-xks-ca" pod "22050016-659c-4232-be33-74bcc81d8535" (UID: "22050016-659c-4232-be33-74bcc81d8535"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:40.001158 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.001124    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m947p\" (UniqueName: \"kubernetes.io/projected/22050016-659c-4232-be33-74bcc81d8535-kube-api-access-m947p\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:40.001158 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.001153    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22050016-659c-4232-be33-74bcc81d8535-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:40.001158 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.001166    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22050016-659c-4232-be33-74bcc81d8535-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:40.375260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.375230    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"
Oct 30 21:25:40.375461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.375269    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9" event={"ID":"22050016-659c-4232-be33-74bcc81d8535","Type":"ContainerDied","Data":"d4d067ef2629949703ecdcd892cae07e9118f26402b72ad53a76368ab48affa9"}
Oct 30 21:25:40.375461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.375317    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4d067ef2629949703ecdcd892cae07e9118f26402b72ad53a76368ab48affa9"
Oct 30 21:25:40.702310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.702227    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"]
Oct 30 21:25:40.703429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.703378    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c77f820a-500a-4201-9fbd-1d470b63971b" containerName="revoke-keys"
Oct 30 21:25:40.703429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.703419    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c77f820a-500a-4201-9fbd-1d470b63971b" containerName="revoke-keys"
Oct 30 21:25:40.703750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.703438    4918 cpu_manager.go:414] "containerMap: removing container" podUID="22050016-659c-4232-be33-74bcc81d8535" containerName="revoke-keys"
Oct 30 21:25:40.703750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.703448    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22050016-659c-4232-be33-74bcc81d8535" containerName="revoke-keys"
Oct 30 21:25:40.703750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.703493    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c75cee27-f8d0-4491-b218-2bda586ed9f8" containerName="revoke-keys"
Oct 30 21:25:40.703750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.703501    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75cee27-f8d0-4491-b218-2bda586ed9f8" containerName="revoke-keys"
Oct 30 21:25:40.703750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.703671    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="22050016-659c-4232-be33-74bcc81d8535" containerName="revoke-keys"
Oct 30 21:25:40.703750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.703699    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75cee27-f8d0-4491-b218-2bda586ed9f8" containerName="revoke-keys"
Oct 30 21:25:40.703750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.703718    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c77f820a-500a-4201-9fbd-1d470b63971b" containerName="revoke-keys"
Oct 30 21:25:40.708084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.708062    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:40.709869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.709851    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"]
Oct 30 21:25:40.722708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.722685    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:40.722917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.722752    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:40.722917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.722823    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:25:40.723024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.722935    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:40.815516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.814130    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaacd451-z9kkb\" (UID: \"10a1378c-4093-4ef9-ac01-a51049248f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:40.815516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.814201    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaacd451-z9kkb\" (UID: \"10a1378c-4093-4ef9-ac01-a51049248f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:40.815516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.814248    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd8mp\" (UniqueName: \"kubernetes.io/projected/10a1378c-4093-4ef9-ac01-a51049248f77-kube-api-access-cd8mp\") pod \"maas-api-revoke-sub-aaacd451-z9kkb\" (UID: \"10a1378c-4093-4ef9-ac01-a51049248f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:40.916967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.916546    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaacd451-z9kkb\" (UID: \"10a1378c-4093-4ef9-ac01-a51049248f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:40.916967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.916609    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaacd451-z9kkb\" (UID: \"10a1378c-4093-4ef9-ac01-a51049248f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:40.916967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.916652    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd8mp\" (UniqueName: \"kubernetes.io/projected/10a1378c-4093-4ef9-ac01-a51049248f77-kube-api-access-cd8mp\") pod \"maas-api-revoke-sub-aaacd451-z9kkb\" (UID: \"10a1378c-4093-4ef9-ac01-a51049248f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:40.917920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.917885    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaacd451-z9kkb\" (UID: \"10a1378c-4093-4ef9-ac01-a51049248f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:40.920279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.920214    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaacd451-z9kkb\" (UID: \"10a1378c-4093-4ef9-ac01-a51049248f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:40.927999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:40.927937    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd8mp\" (UniqueName: \"kubernetes.io/projected/10a1378c-4093-4ef9-ac01-a51049248f77-kube-api-access-cd8mp\") pod \"maas-api-revoke-sub-aaacd451-z9kkb\" (UID: \"10a1378c-4093-4ef9-ac01-a51049248f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:41.021760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.021739    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:25:41.036561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.036538    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.118648    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-token\" (UniqueName: \"kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-token\") pod \"5d77dde2-5153-40d1-8768-1e84c4a210cd\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") "
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.118698    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-credential-socket\") pod \"5d77dde2-5153-40d1-8768-1e84c4a210cd\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") "
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.118856    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d77dde2-5153-40d1-8768-1e84c4a210cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d77dde2-5153-40d1-8768-1e84c4a210cd-istiod-ca-cert\") pod \"5d77dde2-5153-40d1-8768-1e84c4a210cd\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") "
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.118910    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-envoy\") pod \"5d77dde2-5153-40d1-8768-1e84c4a210cd\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") "
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.118962    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-kube-api-access-hjm62\" (UniqueName: \"kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-kube-api-access-hjm62\") pod \"5d77dde2-5153-40d1-8768-1e84c4a210cd\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") "
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.119107    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-socket\") pod \"5d77dde2-5153-40d1-8768-1e84c4a210cd\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") "
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.119137    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-certs\") pod \"5d77dde2-5153-40d1-8768-1e84c4a210cd\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") "
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.119212    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-podinfo\") pod \"5d77dde2-5153-40d1-8768-1e84c4a210cd\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") "
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.119263    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-data\") pod \"5d77dde2-5153-40d1-8768-1e84c4a210cd\" (UID: \"5d77dde2-5153-40d1-8768-1e84c4a210cd\") "
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.119295    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-credential-socket" pod "5d77dde2-5153-40d1-8768-1e84c4a210cd" (UID: "5d77dde2-5153-40d1-8768-1e84c4a210cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.119519    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-socket" pod "5d77dde2-5153-40d1-8768-1e84c4a210cd" (UID: "5d77dde2-5153-40d1-8768-1e84c4a210cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.119672    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-certs" pod "5d77dde2-5153-40d1-8768-1e84c4a210cd" (UID: "5d77dde2-5153-40d1-8768-1e84c4a210cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:41.119714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.119693    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-data" pod "5d77dde2-5153-40d1-8768-1e84c4a210cd" (UID: "5d77dde2-5153-40d1-8768-1e84c4a210cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:41.120699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.119804    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d77dde2-5153-40d1-8768-1e84c4a210cd-istiod-ca-cert" pod "5d77dde2-5153-40d1-8768-1e84c4a210cd" (UID: "5d77dde2-5153-40d1-8768-1e84c4a210cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:41.120699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.119964    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:41.120699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.119985    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d77dde2-5153-40d1-8768-1e84c4a210cd-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:41.120699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.119998    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:41.120699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.120014    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:41.120699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.120027    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:41.123024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.122995    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-podinfo" pod "5d77dde2-5153-40d1-8768-1e84c4a210cd" (UID: "5d77dde2-5153-40d1-8768-1e84c4a210cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:25:41.124183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.124131    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-envoy" pod "5d77dde2-5153-40d1-8768-1e84c4a210cd" (UID: "5d77dde2-5153-40d1-8768-1e84c4a210cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:41.124343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.124321    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-token" pod "5d77dde2-5153-40d1-8768-1e84c4a210cd" (UID: "5d77dde2-5153-40d1-8768-1e84c4a210cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:41.125218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.125192    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-kube-api-access-hjm62" pod "5d77dde2-5153-40d1-8768-1e84c4a210cd" (UID: "5d77dde2-5153-40d1-8768-1e84c4a210cd"). InnerVolumeSpecName "kube-api-access-hjm62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:41.221495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.221464    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:41.221495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.221494    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjm62\" (UniqueName: \"kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-kube-api-access-hjm62\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:41.221711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.221506    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:41.221711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.221514    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d77dde2-5153-40d1-8768-1e84c4a210cd-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:41.323021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.322992    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"]
Oct 30 21:25:41.323156 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:25:41.323098    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a1378c_4093_4ef9_ac01_a51049248f77.slice/crio-792b557523f30508534fa09ff0796d80e26162c89c21dedcac7990068079feb4 WatchSource:0}: Error finding container 792b557523f30508534fa09ff0796d80e26162c89c21dedcac7990068079feb4: Status 404 returned error can't find the container with id 792b557523f30508534fa09ff0796d80e26162c89c21dedcac7990068079feb4
Oct 30 21:25:41.385549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.385514    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb" event={"ID":"10a1378c-4093-4ef9-ac01-a51049248f77","Type":"ContainerStarted","Data":"792b557523f30508534fa09ff0796d80e26162c89c21dedcac7990068079feb4"}
Oct 30 21:25:41.387315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.387289    4918 generic.go:356] "Generic (PLEG): container finished" podID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerID="119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7" exitCode=0
Oct 30 21:25:41.387453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.387349    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"
Oct 30 21:25:41.387453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.387376    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" event={"ID":"5d77dde2-5153-40d1-8768-1e84c4a210cd","Type":"ContainerDied","Data":"119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7"}
Oct 30 21:25:41.387553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.387478    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4" event={"ID":"5d77dde2-5153-40d1-8768-1e84c4a210cd","Type":"ContainerDied","Data":"5957b16a9cc93b210f2d602367ab24d05dc5a62bcc23ead9eda08fe49e5dc03a"}
Oct 30 21:25:41.387553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.387512    4918 scope.go:122] "RemoveContainer" containerID="119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7"
Oct 30 21:25:41.399565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.399546    4918 scope.go:122] "RemoveContainer" containerID="a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce"
Oct 30 21:25:41.410127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.410105    4918 scope.go:122] "RemoveContainer" containerID="119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7"
Oct 30 21:25:41.410447 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:25:41.410425    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7\": container with ID starting with 119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7 not found: ID does not exist" containerID="119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7"
Oct 30 21:25:41.410606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.410456    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7"} err="failed to get container status \"119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7\": rpc error: code = NotFound desc = could not find container \"119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7\": container with ID starting with 119d3269b386da149ff163f2ee523bfc7d580f673ce5fd9d8b7456aea6250de7 not found: ID does not exist"
Oct 30 21:25:41.410606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.410483    4918 scope.go:122] "RemoveContainer" containerID="a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce"
Oct 30 21:25:41.410801 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:25:41.410776    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce\": container with ID starting with a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce not found: ID does not exist" containerID="a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce"
Oct 30 21:25:41.410844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.410813    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce"} err="failed to get container status \"a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce\": rpc error: code = NotFound desc = could not find container \"a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce\": container with ID starting with a0965bcd112715a2dbcfbf41ef1b35973a9c9c34035c727c98b599e904fe40ce not found: ID does not exist"
Oct 30 21:25:41.414907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.414884    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"]
Oct 30 21:25:41.419177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:41.419159    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-7db332-openshift-default-5bcfd47f7c-bdhw4"]
Oct 30 21:25:42.431816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:42.431779    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb" event={"ID":"10a1378c-4093-4ef9-ac01-a51049248f77","Type":"ContainerStarted","Data":"a859dbb5e81eb3cb25c5c07a4cd587fbe6b2eb319f7a537d770f379ce6eb4d22"}
Oct 30 21:25:42.473428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:42.470773    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb" podStartSLOduration=2.470749859 podStartE2EDuration="2.470749859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:42.459918285 +0000 UTC m=+1455.506477446" watchObservedRunningTime="2026-10-30 21:25:42.470749859 +0000 UTC m=+1455.517309005"
Oct 30 21:25:42.927124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:42.927087    4918 patch_prober.go:29] interesting pod/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:25:42.927300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:42.927144    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" podUID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:25:43.079647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:43.079615    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" path="/var/lib/kubelet/pods/5d77dde2-5153-40d1-8768-1e84c4a210cd/volumes"
Oct 30 21:25:43.468091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:43.468062    4918 generic.go:356] "Generic (PLEG): container finished" podID="10a1378c-4093-4ef9-ac01-a51049248f77" containerID="a859dbb5e81eb3cb25c5c07a4cd587fbe6b2eb319f7a537d770f379ce6eb4d22" exitCode=0
Oct 30 21:25:43.468523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:43.468128    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb" event={"ID":"10a1378c-4093-4ef9-ac01-a51049248f77","Type":"ContainerDied","Data":"a859dbb5e81eb3cb25c5c07a4cd587fbe6b2eb319f7a537d770f379ce6eb4d22"}
Oct 30 21:25:44.133186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.133156    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:25:44.176537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.176478    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-socket\") pod \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") "
Oct 30 21:25:44.176847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.176821    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-envoy\") pod \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") "
Oct 30 21:25:44.177082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.177063    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-certs\") pod \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") "
Oct 30 21:25:44.178074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.177292    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istiod-ca-cert\") pod \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") "
Oct 30 21:25:44.183400 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.183346    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-podinfo\") pod \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") "
Oct 30 21:25:44.183724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.183700    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-data\") pod \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") "
Oct 30 21:25:44.183955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.183923    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-certs" pod "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" (UID: "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:44.184259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.184159    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-kube-api-access-zkxxp\" (UniqueName: \"kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-kube-api-access-zkxxp\") pod \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") "
Oct 30 21:25:44.185409 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.185370    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-credential-socket\") pod \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") "
Oct 30 21:25:44.185597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.185578    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-token\" (UniqueName: \"kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-token\") pod \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\" (UID: \"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9\") "
Oct 30 21:25:44.187659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.187611    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-socket" pod "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" (UID: "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:44.187785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.187665    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.189087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.189021    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-data" pod "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" (UID: "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:44.190716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.190689    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istiod-ca-cert" pod "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" (UID: "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:44.194313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.194084    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-credential-socket" pod "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" (UID: "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:44.278078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.278044    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-podinfo" pod "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" (UID: "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:25:44.282538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.282506    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-kube-api-access-zkxxp" pod "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" (UID: "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9"). InnerVolumeSpecName "kube-api-access-zkxxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:44.283601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.283560    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-token" pod "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" (UID: "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:44.288356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.288330    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-envoy" pod "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" (UID: "b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:25:44.293964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.293939    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"]
Oct 30 21:25:44.294861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.294839    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy"
Oct 30 21:25:44.294981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.294863    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy"
Oct 30 21:25:44.294981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.294893    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" containerName="istio-proxy"
Oct 30 21:25:44.294981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.294901    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" containerName="istio-proxy"
Oct 30 21:25:44.294981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.294931    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy"
Oct 30 21:25:44.294981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.294938    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy"
Oct 30 21:25:44.295231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.295117    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy"
Oct 30 21:25:44.295231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.295132    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" containerName="istio-proxy"
Oct 30 21:25:44.295974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.295948    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.296114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.296100    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.296187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.296178    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.296266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.296258    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.296335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.296326    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.296468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.296455    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkxxp\" (UniqueName: \"kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-kube-api-access-zkxxp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.296565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.296554    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.296631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.296623    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.299468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.299448    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:44.301358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.301339    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"]
Oct 30 21:25:44.397864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.397831    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jjqv\" (UniqueName: \"kubernetes.io/projected/8841927f-4cfe-4b94-a720-345ab951fe44-kube-api-access-4jjqv\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj\" (UID: \"8841927f-4cfe-4b94-a720-345ab951fe44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:44.398053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.398032    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj\" (UID: \"8841927f-4cfe-4b94-a720-345ab951fe44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:44.398204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.398189    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj\" (UID: \"8841927f-4cfe-4b94-a720-345ab951fe44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:44.477026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.476999    4918 generic.go:356] "Generic (PLEG): container finished" podID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" containerID="3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910" exitCode=0
Oct 30 21:25:44.477433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.477074    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"
Oct 30 21:25:44.477433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.477083    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" event={"ID":"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9","Type":"ContainerDied","Data":"3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910"}
Oct 30 21:25:44.477433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.477144    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh" event={"ID":"b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9","Type":"ContainerDied","Data":"fa94d2bd78879689fc08c0a66a0f8487e4e58186cae1684f104e45336419d9bb"}
Oct 30 21:25:44.477433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.477176    4918 scope.go:122] "RemoveContainer" containerID="3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910"
Oct 30 21:25:44.489214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.489191    4918 scope.go:122] "RemoveContainer" containerID="3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910"
Oct 30 21:25:44.489568 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:25:44.489540    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910\": container with ID starting with 3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910 not found: ID does not exist" containerID="3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910"
Oct 30 21:25:44.489657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.489579    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910"} err="failed to get container status \"3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910\": rpc error: code = NotFound desc = could not find container \"3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910\": container with ID starting with 3758f5ac20f038e1d29e92e9fff8caacb1a9b5202514c3371ec5017da7c35910 not found: ID does not exist"
Oct 30 21:25:44.502464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.500760    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj\" (UID: \"8841927f-4cfe-4b94-a720-345ab951fe44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:44.502464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.500848    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jjqv\" (UniqueName: \"kubernetes.io/projected/8841927f-4cfe-4b94-a720-345ab951fe44-kube-api-access-4jjqv\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj\" (UID: \"8841927f-4cfe-4b94-a720-345ab951fe44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:44.502464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.500986    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj\" (UID: \"8841927f-4cfe-4b94-a720-345ab951fe44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:44.502464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.501605    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj\" (UID: \"8841927f-4cfe-4b94-a720-345ab951fe44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:44.503970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.503948    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj\" (UID: \"8841927f-4cfe-4b94-a720-345ab951fe44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:44.504111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.504063    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"]
Oct 30 21:25:44.513194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.511773    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-6563b6-openshift-default-5664cf597c-l8nfh"]
Oct 30 21:25:44.513194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.512006    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jjqv\" (UniqueName: \"kubernetes.io/projected/8841927f-4cfe-4b94-a720-345ab951fe44-kube-api-access-4jjqv\") pod \"maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj\" (UID: \"8841927f-4cfe-4b94-a720-345ab951fe44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:44.615452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.615425    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:44.621006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.620984    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:44.706361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.704063    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a1378c-4093-4ef9-ac01-a51049248f77-kube-api-access-cd8mp\" (UniqueName: \"kubernetes.io/projected/10a1378c-4093-4ef9-ac01-a51049248f77-kube-api-access-cd8mp\") pod \"10a1378c-4093-4ef9-ac01-a51049248f77\" (UID: \"10a1378c-4093-4ef9-ac01-a51049248f77\") "
Oct 30 21:25:44.706361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.704116    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-openshift-ca\") pod \"10a1378c-4093-4ef9-ac01-a51049248f77\" (UID: \"10a1378c-4093-4ef9-ac01-a51049248f77\") "
Oct 30 21:25:44.706361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.704173    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-xks-ca\") pod \"10a1378c-4093-4ef9-ac01-a51049248f77\" (UID: \"10a1378c-4093-4ef9-ac01-a51049248f77\") "
Oct 30 21:25:44.707809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.707776    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-openshift-ca" pod "10a1378c-4093-4ef9-ac01-a51049248f77" (UID: "10a1378c-4093-4ef9-ac01-a51049248f77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:44.709689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.709655    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-xks-ca" pod "10a1378c-4093-4ef9-ac01-a51049248f77" (UID: "10a1378c-4093-4ef9-ac01-a51049248f77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:44.711581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.711288    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a1378c-4093-4ef9-ac01-a51049248f77-kube-api-access-cd8mp" pod "10a1378c-4093-4ef9-ac01-a51049248f77" (UID: "10a1378c-4093-4ef9-ac01-a51049248f77"). InnerVolumeSpecName "kube-api-access-cd8mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:44.807242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.807164    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd8mp\" (UniqueName: \"kubernetes.io/projected/10a1378c-4093-4ef9-ac01-a51049248f77-kube-api-access-cd8mp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.807242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.807194    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.807242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.807206    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a1378c-4093-4ef9-ac01-a51049248f77-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:44.920085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:44.920059    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"]
Oct 30 21:25:44.920866 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:25:44.920844    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8841927f_4cfe_4b94_a720_345ab951fe44.slice/crio-174d5a20e54086949eed00044a1b4e7655df607d3b085c38ca2644a16a06e444 WatchSource:0}: Error finding container 174d5a20e54086949eed00044a1b4e7655df607d3b085c38ca2644a16a06e444: Status 404 returned error can't find the container with id 174d5a20e54086949eed00044a1b4e7655df607d3b085c38ca2644a16a06e444
Oct 30 21:25:45.078145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:45.078069    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9" path="/var/lib/kubelet/pods/b876d2d0-7bf1-4fa3-bbc0-3824eb3f8aa9/volumes"
Oct 30 21:25:45.486622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:45.486587    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb" event={"ID":"10a1378c-4093-4ef9-ac01-a51049248f77","Type":"ContainerDied","Data":"792b557523f30508534fa09ff0796d80e26162c89c21dedcac7990068079feb4"}
Oct 30 21:25:45.486622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:45.486619    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"
Oct 30 21:25:45.487101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:45.486643    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="792b557523f30508534fa09ff0796d80e26162c89c21dedcac7990068079feb4"
Oct 30 21:25:45.490463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:45.490438    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj" event={"ID":"8841927f-4cfe-4b94-a720-345ab951fe44","Type":"ContainerStarted","Data":"52880f9bb0801ba2d6d13acd45a12c46d08f4c543fe391a9ed0b58c5c3686c2a"}
Oct 30 21:25:45.490590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:45.490484    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj" event={"ID":"8841927f-4cfe-4b94-a720-345ab951fe44","Type":"ContainerStarted","Data":"174d5a20e54086949eed00044a1b4e7655df607d3b085c38ca2644a16a06e444"}
Oct 30 21:25:45.604508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:45.604441    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj" podStartSLOduration=1.604420414 podStartE2EDuration="1.604420414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:45.552945903 +0000 UTC m=+1458.599505048" watchObservedRunningTime="2026-10-30 21:25:45.604420414 +0000 UTC m=+1458.650979577"
Oct 30 21:25:46.499880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.499845    4918 generic.go:356] "Generic (PLEG): container finished" podID="8841927f-4cfe-4b94-a720-345ab951fe44" containerID="52880f9bb0801ba2d6d13acd45a12c46d08f4c543fe391a9ed0b58c5c3686c2a" exitCode=0
Oct 30 21:25:46.500270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.499942    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj" event={"ID":"8841927f-4cfe-4b94-a720-345ab951fe44","Type":"ContainerDied","Data":"52880f9bb0801ba2d6d13acd45a12c46d08f4c543fe391a9ed0b58c5c3686c2a"}
Oct 30 21:25:46.539568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.539539    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"]
Oct 30 21:25:46.540110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.540094    4918 cpu_manager.go:414] "containerMap: removing container" podUID="10a1378c-4093-4ef9-ac01-a51049248f77" containerName="revoke-keys"
Oct 30 21:25:46.540110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.540108    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a1378c-4093-4ef9-ac01-a51049248f77" containerName="revoke-keys"
Oct 30 21:25:46.540263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.540204    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d77dde2-5153-40d1-8768-1e84c4a210cd" containerName="istio-proxy"
Oct 30 21:25:46.540263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.540211    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a1378c-4093-4ef9-ac01-a51049248f77" containerName="revoke-keys"
Oct 30 21:25:46.543764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.543746    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:46.555496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.555472    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"]
Oct 30 21:25:46.758527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.755950    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4be7be0-8mx4h\" (UID: \"b89002a6-095a-4945-b27d-aa0a51f6fb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:46.758527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.756059    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4be7be0-8mx4h\" (UID: \"b89002a6-095a-4945-b27d-aa0a51f6fb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:46.758527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.756117    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97zg2\" (UniqueName: \"kubernetes.io/projected/b89002a6-095a-4945-b27d-aa0a51f6fb65-kube-api-access-97zg2\") pod \"maas-api-revoke-sub-a4be7be0-8mx4h\" (UID: \"b89002a6-095a-4945-b27d-aa0a51f6fb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:46.857873 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.857841    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4be7be0-8mx4h\" (UID: \"b89002a6-095a-4945-b27d-aa0a51f6fb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:46.858021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.857904    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4be7be0-8mx4h\" (UID: \"b89002a6-095a-4945-b27d-aa0a51f6fb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:46.858021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.857939    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97zg2\" (UniqueName: \"kubernetes.io/projected/b89002a6-095a-4945-b27d-aa0a51f6fb65-kube-api-access-97zg2\") pod \"maas-api-revoke-sub-a4be7be0-8mx4h\" (UID: \"b89002a6-095a-4945-b27d-aa0a51f6fb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:46.858587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.858559    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4be7be0-8mx4h\" (UID: \"b89002a6-095a-4945-b27d-aa0a51f6fb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:46.860342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.860321    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4be7be0-8mx4h\" (UID: \"b89002a6-095a-4945-b27d-aa0a51f6fb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:46.867063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:46.867042    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97zg2\" (UniqueName: \"kubernetes.io/projected/b89002a6-095a-4945-b27d-aa0a51f6fb65-kube-api-access-97zg2\") pod \"maas-api-revoke-sub-a4be7be0-8mx4h\" (UID: \"b89002a6-095a-4945-b27d-aa0a51f6fb65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:47.160366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:47.160288    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:47.516196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:47.516167    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"]
Oct 30 21:25:47.517810 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:25:47.517777    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89002a6_095a_4945_b27d_aa0a51f6fb65.slice/crio-c2d3891fca10cb11d70e0a4d3e58bafa6064bde318dd2f861fb589926c4860c8 WatchSource:0}: Error finding container c2d3891fca10cb11d70e0a4d3e58bafa6064bde318dd2f861fb589926c4860c8: Status 404 returned error can't find the container with id c2d3891fca10cb11d70e0a4d3e58bafa6064bde318dd2f861fb589926c4860c8
Oct 30 21:25:47.636634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:47.636609    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:47.769641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:47.769563    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-openshift-ca\") pod \"8841927f-4cfe-4b94-a720-345ab951fe44\" (UID: \"8841927f-4cfe-4b94-a720-345ab951fe44\") "
Oct 30 21:25:47.769834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:47.769702    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-xks-ca\") pod \"8841927f-4cfe-4b94-a720-345ab951fe44\" (UID: \"8841927f-4cfe-4b94-a720-345ab951fe44\") "
Oct 30 21:25:47.769834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:47.769744    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8841927f-4cfe-4b94-a720-345ab951fe44-kube-api-access-4jjqv\" (UniqueName: \"kubernetes.io/projected/8841927f-4cfe-4b94-a720-345ab951fe44-kube-api-access-4jjqv\") pod \"8841927f-4cfe-4b94-a720-345ab951fe44\" (UID: \"8841927f-4cfe-4b94-a720-345ab951fe44\") "
Oct 30 21:25:47.770005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:47.769977    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-openshift-ca" pod "8841927f-4cfe-4b94-a720-345ab951fe44" (UID: "8841927f-4cfe-4b94-a720-345ab951fe44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:47.770253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:47.770232    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:47.771909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:47.771886    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-xks-ca" pod "8841927f-4cfe-4b94-a720-345ab951fe44" (UID: "8841927f-4cfe-4b94-a720-345ab951fe44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:47.771998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:47.771962    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8841927f-4cfe-4b94-a720-345ab951fe44-kube-api-access-4jjqv" pod "8841927f-4cfe-4b94-a720-345ab951fe44" (UID: "8841927f-4cfe-4b94-a720-345ab951fe44"). InnerVolumeSpecName "kube-api-access-4jjqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:47.872157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:47.872119    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8841927f-4cfe-4b94-a720-345ab951fe44-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:47.872157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:47.872156    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jjqv\" (UniqueName: \"kubernetes.io/projected/8841927f-4cfe-4b94-a720-345ab951fe44-kube-api-access-4jjqv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:48.517647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:48.517617    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"
Oct 30 21:25:48.517647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:48.517630    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj" event={"ID":"8841927f-4cfe-4b94-a720-345ab951fe44","Type":"ContainerDied","Data":"174d5a20e54086949eed00044a1b4e7655df607d3b085c38ca2644a16a06e444"}
Oct 30 21:25:48.518174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:48.517683    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="174d5a20e54086949eed00044a1b4e7655df607d3b085c38ca2644a16a06e444"
Oct 30 21:25:48.519451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:48.519418    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h" event={"ID":"b89002a6-095a-4945-b27d-aa0a51f6fb65","Type":"ContainerStarted","Data":"91e3812fbd7128599a4be44b8380cc5436fbfc70b537807826f7c12dcf3bc3cf"}
Oct 30 21:25:48.519560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:48.519480    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h" event={"ID":"b89002a6-095a-4945-b27d-aa0a51f6fb65","Type":"ContainerStarted","Data":"c2d3891fca10cb11d70e0a4d3e58bafa6064bde318dd2f861fb589926c4860c8"}
Oct 30 21:25:48.537299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:48.537257    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h" podStartSLOduration=2.5372464580000003 podStartE2EDuration="2.537246458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:48.53576777 +0000 UTC m=+1461.582326915" watchObservedRunningTime="2026-10-30 21:25:48.537246458 +0000 UTC m=+1461.583805603"
Oct 30 21:25:49.188308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.188278    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"]
Oct 30 21:25:49.188549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.188512    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9" podUID="6ab512cc-872e-4d6e-a072-9dec56c6b06e" containerName="payload-processing" containerID="cri-o://7b932f0b075c699c7073359514128dca4ead1eb04f7138249ce18e28c6b9aa5e" gracePeriod=30
Oct 30 21:25:49.578055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.578019    4918 generic.go:356] "Generic (PLEG): container finished" podID="6ab512cc-872e-4d6e-a072-9dec56c6b06e" containerID="7b932f0b075c699c7073359514128dca4ead1eb04f7138249ce18e28c6b9aa5e" exitCode=0
Oct 30 21:25:49.578854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.578094    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9" event={"ID":"6ab512cc-872e-4d6e-a072-9dec56c6b06e","Type":"ContainerDied","Data":"7b932f0b075c699c7073359514128dca4ead1eb04f7138249ce18e28c6b9aa5e"}
Oct 30 21:25:49.580249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.580118    4918 generic.go:356] "Generic (PLEG): container finished" podID="b89002a6-095a-4945-b27d-aa0a51f6fb65" containerID="91e3812fbd7128599a4be44b8380cc5436fbfc70b537807826f7c12dcf3bc3cf" exitCode=0
Oct 30 21:25:49.580249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.580198    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h" event={"ID":"b89002a6-095a-4945-b27d-aa0a51f6fb65","Type":"ContainerDied","Data":"91e3812fbd7128599a4be44b8380cc5436fbfc70b537807826f7c12dcf3bc3cf"}
Oct 30 21:25:49.621631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.621579    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"]
Oct 30 21:25:49.632941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.632909    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fde255-tllgn"]
Oct 30 21:25:49.733415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.732835    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"
Oct 30 21:25:49.841103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.835594    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab512cc-872e-4d6e-a072-9dec56c6b06e-kube-api-access-hl94x\" (UniqueName: \"kubernetes.io/projected/6ab512cc-872e-4d6e-a072-9dec56c6b06e-kube-api-access-hl94x\") pod \"6ab512cc-872e-4d6e-a072-9dec56c6b06e\" (UID: \"6ab512cc-872e-4d6e-a072-9dec56c6b06e\") "
Oct 30 21:25:49.841103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.835654    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab512cc-872e-4d6e-a072-9dec56c6b06e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab512cc-872e-4d6e-a072-9dec56c6b06e-plugins-config-volume\") pod \"6ab512cc-872e-4d6e-a072-9dec56c6b06e\" (UID: \"6ab512cc-872e-4d6e-a072-9dec56c6b06e\") "
Oct 30 21:25:49.841103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.837691    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab512cc-872e-4d6e-a072-9dec56c6b06e-plugins-config-volume" pod "6ab512cc-872e-4d6e-a072-9dec56c6b06e" (UID: "6ab512cc-872e-4d6e-a072-9dec56c6b06e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:49.850622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.848654    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab512cc-872e-4d6e-a072-9dec56c6b06e-kube-api-access-hl94x" pod "6ab512cc-872e-4d6e-a072-9dec56c6b06e" (UID: "6ab512cc-872e-4d6e-a072-9dec56c6b06e"). InnerVolumeSpecName "kube-api-access-hl94x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:49.938262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.938234    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl94x\" (UniqueName: \"kubernetes.io/projected/6ab512cc-872e-4d6e-a072-9dec56c6b06e-kube-api-access-hl94x\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:49.938262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:49.938257    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab512cc-872e-4d6e-a072-9dec56c6b06e-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:50.640131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:50.640100    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"
Oct 30 21:25:50.653188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:50.651318    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9" event={"ID":"6ab512cc-872e-4d6e-a072-9dec56c6b06e","Type":"ContainerDied","Data":"afae62df03f003f437c1ea73b7689ef5ad09b2a17cd6de18704224f904c7ddca"}
Oct 30 21:25:50.653188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:50.651419    4918 scope.go:122] "RemoveContainer" containerID="7b932f0b075c699c7073359514128dca4ead1eb04f7138249ce18e28c6b9aa5e"
Oct 30 21:25:50.735578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:50.735528    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"]
Oct 30 21:25:50.740557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:50.740406    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fde255-bf5c5c99d-g5wb9"]
Oct 30 21:25:50.743987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:50.743960    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"]
Oct 30 21:25:50.747739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:50.747712    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-67b92c-gf9h9"]
Oct 30 21:25:50.962570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:50.962548    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:51.057301 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.057274    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b89002a6-095a-4945-b27d-aa0a51f6fb65-kube-api-access-97zg2\" (UniqueName: \"kubernetes.io/projected/b89002a6-095a-4945-b27d-aa0a51f6fb65-kube-api-access-97zg2\") pod \"b89002a6-095a-4945-b27d-aa0a51f6fb65\" (UID: \"b89002a6-095a-4945-b27d-aa0a51f6fb65\") "
Oct 30 21:25:51.057473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.057327    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-openshift-ca\") pod \"b89002a6-095a-4945-b27d-aa0a51f6fb65\" (UID: \"b89002a6-095a-4945-b27d-aa0a51f6fb65\") "
Oct 30 21:25:51.057578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.057481    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-xks-ca\") pod \"b89002a6-095a-4945-b27d-aa0a51f6fb65\" (UID: \"b89002a6-095a-4945-b27d-aa0a51f6fb65\") "
Oct 30 21:25:51.057713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.057693    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-openshift-ca" pod "b89002a6-095a-4945-b27d-aa0a51f6fb65" (UID: "b89002a6-095a-4945-b27d-aa0a51f6fb65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:51.057950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.057931    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:51.059637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.059611    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89002a6-095a-4945-b27d-aa0a51f6fb65-kube-api-access-97zg2" pod "b89002a6-095a-4945-b27d-aa0a51f6fb65" (UID: "b89002a6-095a-4945-b27d-aa0a51f6fb65"). InnerVolumeSpecName "kube-api-access-97zg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:51.059732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.059686    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-xks-ca" pod "b89002a6-095a-4945-b27d-aa0a51f6fb65" (UID: "b89002a6-095a-4945-b27d-aa0a51f6fb65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:51.077918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.077898    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22050016-659c-4232-be33-74bcc81d8535" path="/var/lib/kubelet/pods/22050016-659c-4232-be33-74bcc81d8535/volumes"
Oct 30 21:25:51.078280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.078267    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab512cc-872e-4d6e-a072-9dec56c6b06e" path="/var/lib/kubelet/pods/6ab512cc-872e-4d6e-a072-9dec56c6b06e/volumes"
Oct 30 21:25:51.078670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.078657    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75cee27-f8d0-4491-b218-2bda586ed9f8" path="/var/lib/kubelet/pods/c75cee27-f8d0-4491-b218-2bda586ed9f8/volumes"
Oct 30 21:25:51.159427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.159364    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89002a6-095a-4945-b27d-aa0a51f6fb65-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:51.159427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.159416    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97zg2\" (UniqueName: \"kubernetes.io/projected/b89002a6-095a-4945-b27d-aa0a51f6fb65-kube-api-access-97zg2\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:51.649894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.649868    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"
Oct 30 21:25:51.650304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.649898    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h" event={"ID":"b89002a6-095a-4945-b27d-aa0a51f6fb65","Type":"ContainerDied","Data":"c2d3891fca10cb11d70e0a4d3e58bafa6064bde318dd2f861fb589926c4860c8"}
Oct 30 21:25:51.650304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:51.649944    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2d3891fca10cb11d70e0a4d3e58bafa6064bde318dd2f861fb589926c4860c8"
Oct 30 21:25:52.736484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.726889    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"]
Oct 30 21:25:52.736484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.728436    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6ab512cc-872e-4d6e-a072-9dec56c6b06e" containerName="payload-processing"
Oct 30 21:25:52.736484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.728458    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab512cc-872e-4d6e-a072-9dec56c6b06e" containerName="payload-processing"
Oct 30 21:25:52.736484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.728553    4918 cpu_manager.go:414] "containerMap: removing container" podUID="8841927f-4cfe-4b94-a720-345ab951fe44" containerName="revoke-keys"
Oct 30 21:25:52.736484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.728563    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8841927f-4cfe-4b94-a720-345ab951fe44" containerName="revoke-keys"
Oct 30 21:25:52.736484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.728581    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b89002a6-095a-4945-b27d-aa0a51f6fb65" containerName="revoke-keys"
Oct 30 21:25:52.736484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.728589    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b89002a6-095a-4945-b27d-aa0a51f6fb65" containerName="revoke-keys"
Oct 30 21:25:52.736484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.728758    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8841927f-4cfe-4b94-a720-345ab951fe44" containerName="revoke-keys"
Oct 30 21:25:52.736484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.728773    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b89002a6-095a-4945-b27d-aa0a51f6fb65" containerName="revoke-keys"
Oct 30 21:25:52.736484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.728787    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab512cc-872e-4d6e-a072-9dec56c6b06e" containerName="payload-processing"
Oct 30 21:25:52.738627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.738603    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:52.740115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.740030    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"]
Oct 30 21:25:52.742863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.742836    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:25:52.743692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.743671    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:25:52.780945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.780904    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:25:52.781083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.780981    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:25:52.795910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.795882    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440077d5-xvjfp\" (UID: \"5c4ead3c-02e4-48a7-8a50-5c090b997489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:52.796067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.795926    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440077d5-xvjfp\" (UID: \"5c4ead3c-02e4-48a7-8a50-5c090b997489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:52.796067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.796029    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4dtw\" (UniqueName: \"kubernetes.io/projected/5c4ead3c-02e4-48a7-8a50-5c090b997489-kube-api-access-v4dtw\") pod \"maas-api-revoke-sub-440077d5-xvjfp\" (UID: \"5c4ead3c-02e4-48a7-8a50-5c090b997489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:52.896853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.896821    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4dtw\" (UniqueName: \"kubernetes.io/projected/5c4ead3c-02e4-48a7-8a50-5c090b997489-kube-api-access-v4dtw\") pod \"maas-api-revoke-sub-440077d5-xvjfp\" (UID: \"5c4ead3c-02e4-48a7-8a50-5c090b997489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:52.897028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.896998    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440077d5-xvjfp\" (UID: \"5c4ead3c-02e4-48a7-8a50-5c090b997489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:52.897079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.897032    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440077d5-xvjfp\" (UID: \"5c4ead3c-02e4-48a7-8a50-5c090b997489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:52.897778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.897754    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440077d5-xvjfp\" (UID: \"5c4ead3c-02e4-48a7-8a50-5c090b997489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:52.899667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.899643    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440077d5-xvjfp\" (UID: \"5c4ead3c-02e4-48a7-8a50-5c090b997489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:52.911909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:52.911885    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4dtw\" (UniqueName: \"kubernetes.io/projected/5c4ead3c-02e4-48a7-8a50-5c090b997489-kube-api-access-v4dtw\") pod \"maas-api-revoke-sub-440077d5-xvjfp\" (UID: \"5c4ead3c-02e4-48a7-8a50-5c090b997489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:53.065643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:53.065566    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:53.345025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:53.345000    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"]
Oct 30 21:25:53.345879 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:25:53.345854    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4ead3c_02e4_48a7_8a50_5c090b997489.slice/crio-420341ac618312bb3aa6d47035a37e21cce4e34e4e8f282082a613a4fc0b9f95 WatchSource:0}: Error finding container 420341ac618312bb3aa6d47035a37e21cce4e34e4e8f282082a613a4fc0b9f95: Status 404 returned error can't find the container with id 420341ac618312bb3aa6d47035a37e21cce4e34e4e8f282082a613a4fc0b9f95
Oct 30 21:25:53.684120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:53.684036    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp" event={"ID":"5c4ead3c-02e4-48a7-8a50-5c090b997489","Type":"ContainerStarted","Data":"420341ac618312bb3aa6d47035a37e21cce4e34e4e8f282082a613a4fc0b9f95"}
Oct 30 21:25:54.719885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:54.719799    4918 generic.go:356] "Generic (PLEG): container finished" podID="5c4ead3c-02e4-48a7-8a50-5c090b997489" containerID="a6da856b6503ec975f18090a67fc73fb3ea26e6588e0572f0cb3e62fb49523db" exitCode=0
Oct 30 21:25:54.721034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:54.720842    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp" event={"ID":"5c4ead3c-02e4-48a7-8a50-5c090b997489","Type":"ContainerDied","Data":"a6da856b6503ec975f18090a67fc73fb3ea26e6588e0572f0cb3e62fb49523db"}
Oct 30 21:25:56.053509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.053484    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:56.193485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.193455    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-xks-ca\") pod \"5c4ead3c-02e4-48a7-8a50-5c090b997489\" (UID: \"5c4ead3c-02e4-48a7-8a50-5c090b997489\") "
Oct 30 21:25:56.193668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.193601    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-openshift-ca\") pod \"5c4ead3c-02e4-48a7-8a50-5c090b997489\" (UID: \"5c4ead3c-02e4-48a7-8a50-5c090b997489\") "
Oct 30 21:25:56.193886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.193863    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4ead3c-02e4-48a7-8a50-5c090b997489-kube-api-access-v4dtw\" (UniqueName: \"kubernetes.io/projected/5c4ead3c-02e4-48a7-8a50-5c090b997489-kube-api-access-v4dtw\") pod \"5c4ead3c-02e4-48a7-8a50-5c090b997489\" (UID: \"5c4ead3c-02e4-48a7-8a50-5c090b997489\") "
Oct 30 21:25:56.193996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.193969    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-openshift-ca" pod "5c4ead3c-02e4-48a7-8a50-5c090b997489" (UID: "5c4ead3c-02e4-48a7-8a50-5c090b997489"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:25:56.195456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.195204    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:56.196198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.196171    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-xks-ca" pod "5c4ead3c-02e4-48a7-8a50-5c090b997489" (UID: "5c4ead3c-02e4-48a7-8a50-5c090b997489"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:25:56.196320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.196256    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4ead3c-02e4-48a7-8a50-5c090b997489-kube-api-access-v4dtw" pod "5c4ead3c-02e4-48a7-8a50-5c090b997489" (UID: "5c4ead3c-02e4-48a7-8a50-5c090b997489"). InnerVolumeSpecName "kube-api-access-v4dtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:25:56.299743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.298651    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4dtw\" (UniqueName: \"kubernetes.io/projected/5c4ead3c-02e4-48a7-8a50-5c090b997489-kube-api-access-v4dtw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:56.299743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.298696    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4ead3c-02e4-48a7-8a50-5c090b997489-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:25:56.737692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.737667    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"
Oct 30 21:25:56.737888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.737700    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp" event={"ID":"5c4ead3c-02e4-48a7-8a50-5c090b997489","Type":"ContainerDied","Data":"420341ac618312bb3aa6d47035a37e21cce4e34e4e8f282082a613a4fc0b9f95"}
Oct 30 21:25:56.737888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:56.737776    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="420341ac618312bb3aa6d47035a37e21cce4e34e4e8f282082a613a4fc0b9f95"
Oct 30 21:25:57.082321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:57.082249    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"]
Oct 30 21:25:57.082786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:57.082452    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy" containerID="cri-o://71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419" gracePeriod=30
Oct 30 21:25:58.065153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:58.064994    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"]
Oct 30 21:25:58.065923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:58.065251    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerName="istio-proxy" containerID="cri-o://81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4" gracePeriod=30
Oct 30 21:25:59.161367 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:25:59.161333    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:25:59.496887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:59.496818    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"]
Oct 30 21:25:59.518216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:25:59.518184    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-50d070-s86sj"]
Oct 30 21:26:01.129491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:01.129453    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8841927f-4cfe-4b94-a720-345ab951fe44" path="/var/lib/kubelet/pods/8841927f-4cfe-4b94-a720-345ab951fe44/volumes"
Oct 30 21:26:02.382575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.382458    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:02.684367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.684342    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:26:02.718641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.718607    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-credential-socket\") pod \"46b9e248-61d3-4cf1-823d-da0343cb6dec\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") "
Oct 30 21:26:02.719227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.719197    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-credential-socket" pod "46b9e248-61d3-4cf1-823d-da0343cb6dec" (UID: "46b9e248-61d3-4cf1-823d-da0343cb6dec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:02.719350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.719280    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b9e248-61d3-4cf1-823d-da0343cb6dec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b9e248-61d3-4cf1-823d-da0343cb6dec-istiod-ca-cert\") pod \"46b9e248-61d3-4cf1-823d-da0343cb6dec\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") "
Oct 30 21:26:02.719350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.719331    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-podinfo\") pod \"46b9e248-61d3-4cf1-823d-da0343cb6dec\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") "
Oct 30 21:26:02.719521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.719373    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-certs\") pod \"46b9e248-61d3-4cf1-823d-da0343cb6dec\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") "
Oct 30 21:26:02.719521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.719420    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-token\" (UniqueName: \"kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-token\") pod \"46b9e248-61d3-4cf1-823d-da0343cb6dec\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") "
Oct 30 21:26:02.719628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.719543    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-socket\") pod \"46b9e248-61d3-4cf1-823d-da0343cb6dec\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") "
Oct 30 21:26:02.719628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.719619    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-envoy\") pod \"46b9e248-61d3-4cf1-823d-da0343cb6dec\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") "
Oct 30 21:26:02.719736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.719680    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-data\") pod \"46b9e248-61d3-4cf1-823d-da0343cb6dec\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") "
Oct 30 21:26:02.719854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.719835    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-kube-api-access-x8r6n\" (UniqueName: \"kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-kube-api-access-x8r6n\") pod \"46b9e248-61d3-4cf1-823d-da0343cb6dec\" (UID: \"46b9e248-61d3-4cf1-823d-da0343cb6dec\") "
Oct 30 21:26:02.723340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.723310    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-podinfo" pod "46b9e248-61d3-4cf1-823d-da0343cb6dec" (UID: "46b9e248-61d3-4cf1-823d-da0343cb6dec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:26:02.723554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.723522    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-certs" pod "46b9e248-61d3-4cf1-823d-da0343cb6dec" (UID: "46b9e248-61d3-4cf1-823d-da0343cb6dec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:02.723770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.723746    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b9e248-61d3-4cf1-823d-da0343cb6dec-istiod-ca-cert" pod "46b9e248-61d3-4cf1-823d-da0343cb6dec" (UID: "46b9e248-61d3-4cf1-823d-da0343cb6dec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:02.723864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.723841    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-socket" pod "46b9e248-61d3-4cf1-823d-da0343cb6dec" (UID: "46b9e248-61d3-4cf1-823d-da0343cb6dec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:02.724054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.724031    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-token" pod "46b9e248-61d3-4cf1-823d-da0343cb6dec" (UID: "46b9e248-61d3-4cf1-823d-da0343cb6dec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:02.724216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.724188    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-data" pod "46b9e248-61d3-4cf1-823d-da0343cb6dec" (UID: "46b9e248-61d3-4cf1-823d-da0343cb6dec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:02.724457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.724433    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:02.724688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.724460    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:02.724688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.724476    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:02.724688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.724491    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b9e248-61d3-4cf1-823d-da0343cb6dec-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:02.724688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.724505    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:02.724688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.724519    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:02.724688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.724533    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:02.725793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.725764    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-envoy" pod "46b9e248-61d3-4cf1-823d-da0343cb6dec" (UID: "46b9e248-61d3-4cf1-823d-da0343cb6dec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:02.727441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.727414    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-kube-api-access-x8r6n" pod "46b9e248-61d3-4cf1-823d-da0343cb6dec" (UID: "46b9e248-61d3-4cf1-823d-da0343cb6dec"). InnerVolumeSpecName "kube-api-access-x8r6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:02.825670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.825641    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b9e248-61d3-4cf1-823d-da0343cb6dec-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:02.825670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.825667    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8r6n\" (UniqueName: \"kubernetes.io/projected/46b9e248-61d3-4cf1-823d-da0343cb6dec-kube-api-access-x8r6n\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:02.850225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.850193    4918 generic.go:356] "Generic (PLEG): container finished" podID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerID="71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419" exitCode=0
Oct 30 21:26:02.850376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.850245    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"
Oct 30 21:26:02.850376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.850284    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" event={"ID":"46b9e248-61d3-4cf1-823d-da0343cb6dec","Type":"ContainerDied","Data":"71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419"}
Oct 30 21:26:02.850376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.850342    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg" event={"ID":"46b9e248-61d3-4cf1-823d-da0343cb6dec","Type":"ContainerDied","Data":"dc6936a54597eac3b9d7f25a1a88f8d3a294f86f08c410f18ee783dfd5e63a42"}
Oct 30 21:26:02.850376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.850370    4918 scope.go:122] "RemoveContainer" containerID="71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419"
Oct 30 21:26:02.862088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.862070    4918 scope.go:122] "RemoveContainer" containerID="9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6"
Oct 30 21:26:02.872500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.872436    4918 scope.go:122] "RemoveContainer" containerID="71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419"
Oct 30 21:26:02.872751 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:26:02.872731    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419\": container with ID starting with 71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419 not found: ID does not exist" containerID="71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419"
Oct 30 21:26:02.872819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.872760    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419"} err="failed to get container status \"71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419\": rpc error: code = NotFound desc = could not find container \"71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419\": container with ID starting with 71e2104a2f7fcffe7c05fbada7c29bbf285110aec3f7a66f3c184b2fb3ff5419 not found: ID does not exist"
Oct 30 21:26:02.872819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.872775    4918 scope.go:122] "RemoveContainer" containerID="9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6"
Oct 30 21:26:02.873027 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:26:02.873013    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6\": container with ID starting with 9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6 not found: ID does not exist" containerID="9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6"
Oct 30 21:26:02.873081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.873032    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6"} err="failed to get container status \"9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6\": rpc error: code = NotFound desc = could not find container \"9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6\": container with ID starting with 9bd52fd447333a58d44d714e37189fe1068f6d0a489e9045bc7cfda4cfa9f4f6 not found: ID does not exist"
Oct 30 21:26:02.875851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.875830    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"]
Oct 30 21:26:02.880517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:02.880499    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-fde255-openshift-default-7bc4bf874c-557jg"]
Oct 30 21:26:03.079350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.079325    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" path="/var/lib/kubelet/pods/46b9e248-61d3-4cf1-823d-da0343cb6dec/volumes"
Oct 30 21:26:03.339195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.339138    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:26:03.434149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.433777    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-socket\") pod \"a63cdd9a-24cd-4674-b165-a21d9781f974\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") "
Oct 30 21:26:03.434149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.433927    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63cdd9a-24cd-4674-b165-a21d9781f974-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cdd9a-24cd-4674-b165-a21d9781f974-istiod-ca-cert\") pod \"a63cdd9a-24cd-4674-b165-a21d9781f974\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") "
Oct 30 21:26:03.469458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.468704    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-socket" pod "a63cdd9a-24cd-4674-b165-a21d9781f974" (UID: "a63cdd9a-24cd-4674-b165-a21d9781f974"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:03.469458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.468868    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-token\" (UniqueName: \"kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-token\") pod \"a63cdd9a-24cd-4674-b165-a21d9781f974\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") "
Oct 30 21:26:03.469458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.468932    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-certs\") pod \"a63cdd9a-24cd-4674-b165-a21d9781f974\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") "
Oct 30 21:26:03.469458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.469001    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-envoy\") pod \"a63cdd9a-24cd-4674-b165-a21d9781f974\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") "
Oct 30 21:26:03.469458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.469114    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-credential-socket\") pod \"a63cdd9a-24cd-4674-b165-a21d9781f974\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") "
Oct 30 21:26:03.469458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.469153    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-kube-api-access-8p2f5\" (UniqueName: \"kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-kube-api-access-8p2f5\") pod \"a63cdd9a-24cd-4674-b165-a21d9781f974\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") "
Oct 30 21:26:03.469458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.469228    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-data\") pod \"a63cdd9a-24cd-4674-b165-a21d9781f974\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") "
Oct 30 21:26:03.469458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.469266    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-podinfo\") pod \"a63cdd9a-24cd-4674-b165-a21d9781f974\" (UID: \"a63cdd9a-24cd-4674-b165-a21d9781f974\") "
Oct 30 21:26:03.470521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.470128    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.470521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.470192    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63cdd9a-24cd-4674-b165-a21d9781f974-istiod-ca-cert" pod "a63cdd9a-24cd-4674-b165-a21d9781f974" (UID: "a63cdd9a-24cd-4674-b165-a21d9781f974"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:03.471757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.471671    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-credential-socket" pod "a63cdd9a-24cd-4674-b165-a21d9781f974" (UID: "a63cdd9a-24cd-4674-b165-a21d9781f974"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:03.471757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.471731    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-data" pod "a63cdd9a-24cd-4674-b165-a21d9781f974" (UID: "a63cdd9a-24cd-4674-b165-a21d9781f974"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:03.472006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.471972    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-certs" pod "a63cdd9a-24cd-4674-b165-a21d9781f974" (UID: "a63cdd9a-24cd-4674-b165-a21d9781f974"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:03.480612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.480579    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-envoy" pod "a63cdd9a-24cd-4674-b165-a21d9781f974" (UID: "a63cdd9a-24cd-4674-b165-a21d9781f974"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:26:03.480756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.480669    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-kube-api-access-8p2f5" pod "a63cdd9a-24cd-4674-b165-a21d9781f974" (UID: "a63cdd9a-24cd-4674-b165-a21d9781f974"). InnerVolumeSpecName "kube-api-access-8p2f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:03.482932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.482907    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-podinfo" pod "a63cdd9a-24cd-4674-b165-a21d9781f974" (UID: "a63cdd9a-24cd-4674-b165-a21d9781f974"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:26:03.483199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.483180    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-token" pod "a63cdd9a-24cd-4674-b165-a21d9781f974" (UID: "a63cdd9a-24cd-4674-b165-a21d9781f974"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:03.571532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.571497    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.571532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.571523    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.571532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.571533    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.571795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.571542    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p2f5\" (UniqueName: \"kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-kube-api-access-8p2f5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.571795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.571552    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.571795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.571560    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.571795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.571568    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cdd9a-24cd-4674-b165-a21d9781f974-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.571795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.571576    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63cdd9a-24cd-4674-b165-a21d9781f974-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:03.860929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.860894    4918 generic.go:356] "Generic (PLEG): container finished" podID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerID="81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4" exitCode=0
Oct 30 21:26:03.861086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.860967    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"
Oct 30 21:26:03.861086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.860978    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" event={"ID":"a63cdd9a-24cd-4674-b165-a21d9781f974","Type":"ContainerDied","Data":"81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4"}
Oct 30 21:26:03.861086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.861056    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77" event={"ID":"a63cdd9a-24cd-4674-b165-a21d9781f974","Type":"ContainerDied","Data":"9dbe2cf3dc7375cf5a48514966abd607325efe28fdaf5c5d09eb7d4e123394bc"}
Oct 30 21:26:03.861086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.861080    4918 scope.go:122] "RemoveContainer" containerID="81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4"
Oct 30 21:26:03.872994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.872976    4918 scope.go:122] "RemoveContainer" containerID="2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223"
Oct 30 21:26:03.883554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.883537    4918 scope.go:122] "RemoveContainer" containerID="81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4"
Oct 30 21:26:03.883828 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:26:03.883806    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4\": container with ID starting with 81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4 not found: ID does not exist" containerID="81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4"
Oct 30 21:26:03.883891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.883836    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4"} err="failed to get container status \"81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4\": rpc error: code = NotFound desc = could not find container \"81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4\": container with ID starting with 81b30d633b03cd2983c7a50b115755e87cebe10ef0e3e80dd789c83d8e2934a4 not found: ID does not exist"
Oct 30 21:26:03.883891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.883852    4918 scope.go:122] "RemoveContainer" containerID="2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223"
Oct 30 21:26:03.884144 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:26:03.884121    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223\": container with ID starting with 2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223 not found: ID does not exist" containerID="2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223"
Oct 30 21:26:03.884198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.884154    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223"} err="failed to get container status \"2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223\": rpc error: code = NotFound desc = could not find container \"2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223\": container with ID starting with 2dcb53d29e35adc3955aa5d7f937607215b70aeb16a1c9393deffda66be8e223 not found: ID does not exist"
Oct 30 21:26:03.886716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.886689    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"]
Oct 30 21:26:03.890774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:03.890755    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-67b92c-openshift-default-6445c8c557-z7c77"]
Oct 30 21:26:04.208157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208082    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"]
Oct 30 21:26:04.208694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208678    4918 cpu_manager.go:414] "containerMap: removing container" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy"
Oct 30 21:26:04.208759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208696    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy"
Oct 30 21:26:04.208759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208710    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerName="istio-proxy"
Oct 30 21:26:04.208759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208716    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerName="istio-proxy"
Oct 30 21:26:04.208759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208731    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerName="istio-proxy"
Oct 30 21:26:04.208759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208736    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerName="istio-proxy"
Oct 30 21:26:04.208759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208757    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5c4ead3c-02e4-48a7-8a50-5c090b997489" containerName="revoke-keys"
Oct 30 21:26:04.208951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208762    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c4ead3c-02e4-48a7-8a50-5c090b997489" containerName="revoke-keys"
Oct 30 21:26:04.208951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208842    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy"
Oct 30 21:26:04.208951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208851    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerName="istio-proxy"
Oct 30 21:26:04.208951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208860    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" containerName="istio-proxy"
Oct 30 21:26:04.208951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.208867    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c4ead3c-02e4-48a7-8a50-5c090b997489" containerName="revoke-keys"
Oct 30 21:26:04.213938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.213919    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:04.218233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.218209    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:04.218361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.218286    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:04.218361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.218325    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:04.219035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.219019    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:26:04.220799 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.220780    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"]
Oct 30 21:26:04.288421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.285288    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjm4k\" (UniqueName: \"kubernetes.io/projected/d66548e1-39b4-40d8-a84b-c91df71a5726-kube-api-access-mjm4k\") pod \"maas-api-revoke-sub-6152c89d-tzscz\" (UID: \"d66548e1-39b4-40d8-a84b-c91df71a5726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:04.288421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.285352    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6152c89d-tzscz\" (UID: \"d66548e1-39b4-40d8-a84b-c91df71a5726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:04.288421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.285548    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6152c89d-tzscz\" (UID: \"d66548e1-39b4-40d8-a84b-c91df71a5726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:04.386510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.386475    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjm4k\" (UniqueName: \"kubernetes.io/projected/d66548e1-39b4-40d8-a84b-c91df71a5726-kube-api-access-mjm4k\") pod \"maas-api-revoke-sub-6152c89d-tzscz\" (UID: \"d66548e1-39b4-40d8-a84b-c91df71a5726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:04.386510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.386512    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6152c89d-tzscz\" (UID: \"d66548e1-39b4-40d8-a84b-c91df71a5726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:04.386735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.386617    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6152c89d-tzscz\" (UID: \"d66548e1-39b4-40d8-a84b-c91df71a5726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:04.387292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.387273    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6152c89d-tzscz\" (UID: \"d66548e1-39b4-40d8-a84b-c91df71a5726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:04.389096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.389073    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6152c89d-tzscz\" (UID: \"d66548e1-39b4-40d8-a84b-c91df71a5726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:04.395892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.395870    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjm4k\" (UniqueName: \"kubernetes.io/projected/d66548e1-39b4-40d8-a84b-c91df71a5726-kube-api-access-mjm4k\") pod \"maas-api-revoke-sub-6152c89d-tzscz\" (UID: \"d66548e1-39b4-40d8-a84b-c91df71a5726\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:04.531630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.531608    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:04.812070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.812033    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"]
Oct 30 21:26:04.812755 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:04.812727    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66548e1_39b4_40d8_a84b_c91df71a5726.slice/crio-ac1fc43b3f5699a0834162333ecdcd584a32e5fc29f7183226cf3ab43e93eb3c WatchSource:0}: Error finding container ac1fc43b3f5699a0834162333ecdcd584a32e5fc29f7183226cf3ab43e93eb3c: Status 404 returned error can't find the container with id ac1fc43b3f5699a0834162333ecdcd584a32e5fc29f7183226cf3ab43e93eb3c
Oct 30 21:26:04.871927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:04.871900    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz" event={"ID":"d66548e1-39b4-40d8-a84b-c91df71a5726","Type":"ContainerStarted","Data":"ac1fc43b3f5699a0834162333ecdcd584a32e5fc29f7183226cf3ab43e93eb3c"}
Oct 30 21:26:05.078210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:05.078138    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63cdd9a-24cd-4674-b165-a21d9781f974" path="/var/lib/kubelet/pods/a63cdd9a-24cd-4674-b165-a21d9781f974/volumes"
Oct 30 21:26:05.919626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:05.919451    4918 generic.go:356] "Generic (PLEG): container finished" podID="d66548e1-39b4-40d8-a84b-c91df71a5726" containerID="f63107ee6a91530d607ab6069a19046e44c761364617dc7288356f4443c6815d" exitCode=0
Oct 30 21:26:05.919626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:05.919542    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz" event={"ID":"d66548e1-39b4-40d8-a84b-c91df71a5726","Type":"ContainerDied","Data":"f63107ee6a91530d607ab6069a19046e44c761364617dc7288356f4443c6815d"}
Oct 30 21:26:07.098368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.098345    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:07.241676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.241633    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-xks-ca\") pod \"d66548e1-39b4-40d8-a84b-c91df71a5726\" (UID: \"d66548e1-39b4-40d8-a84b-c91df71a5726\") "
Oct 30 21:26:07.241676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.241667    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-openshift-ca\") pod \"d66548e1-39b4-40d8-a84b-c91df71a5726\" (UID: \"d66548e1-39b4-40d8-a84b-c91df71a5726\") "
Oct 30 21:26:07.242234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.241837    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d66548e1-39b4-40d8-a84b-c91df71a5726-kube-api-access-mjm4k\" (UniqueName: \"kubernetes.io/projected/d66548e1-39b4-40d8-a84b-c91df71a5726-kube-api-access-mjm4k\") pod \"d66548e1-39b4-40d8-a84b-c91df71a5726\" (UID: \"d66548e1-39b4-40d8-a84b-c91df71a5726\") "
Oct 30 21:26:07.242234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.242139    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-openshift-ca" pod "d66548e1-39b4-40d8-a84b-c91df71a5726" (UID: "d66548e1-39b4-40d8-a84b-c91df71a5726"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:07.242505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.242477    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:07.243994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.243967    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-xks-ca" pod "d66548e1-39b4-40d8-a84b-c91df71a5726" (UID: "d66548e1-39b4-40d8-a84b-c91df71a5726"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:07.244110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.244088    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66548e1-39b4-40d8-a84b-c91df71a5726-kube-api-access-mjm4k" pod "d66548e1-39b4-40d8-a84b-c91df71a5726" (UID: "d66548e1-39b4-40d8-a84b-c91df71a5726"). InnerVolumeSpecName "kube-api-access-mjm4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:07.344146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.344112    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjm4k\" (UniqueName: \"kubernetes.io/projected/d66548e1-39b4-40d8-a84b-c91df71a5726-kube-api-access-mjm4k\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:07.344146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.344142    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d66548e1-39b4-40d8-a84b-c91df71a5726-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:07.935812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.935783    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"
Oct 30 21:26:07.935983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.935819    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz" event={"ID":"d66548e1-39b4-40d8-a84b-c91df71a5726","Type":"ContainerDied","Data":"ac1fc43b3f5699a0834162333ecdcd584a32e5fc29f7183226cf3ab43e93eb3c"}
Oct 30 21:26:07.935983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:07.935895    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac1fc43b3f5699a0834162333ecdcd584a32e5fc29f7183226cf3ab43e93eb3c"
Oct 30 21:26:08.059431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.059379    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"]
Oct 30 21:26:08.059935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.059920    4918 cpu_manager.go:414] "containerMap: removing container" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy"
Oct 30 21:26:08.059996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.059936    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy"
Oct 30 21:26:08.059996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.059965    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d66548e1-39b4-40d8-a84b-c91df71a5726" containerName="revoke-keys"
Oct 30 21:26:08.059996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.059970    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d66548e1-39b4-40d8-a84b-c91df71a5726" containerName="revoke-keys"
Oct 30 21:26:08.060098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.060068    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b9e248-61d3-4cf1-823d-da0343cb6dec" containerName="istio-proxy"
Oct 30 21:26:08.060098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.060080    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d66548e1-39b4-40d8-a84b-c91df71a5726" containerName="revoke-keys"
Oct 30 21:26:08.063738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.063722    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:08.072180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.072159    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:08.072469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.072426    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:26:08.072868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.072849    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:08.081731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.081704    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"]
Oct 30 21:26:08.116007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.115983    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:08.153726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.153697    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzmx\" (UniqueName: \"kubernetes.io/projected/e194051c-28aa-4275-8a63-ce4ce433e23d-kube-api-access-cnzmx\") pod \"maas-api-revoke-sub-f1f84cce-9t7jr\" (UID: \"e194051c-28aa-4275-8a63-ce4ce433e23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:08.153915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.153775    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f84cce-9t7jr\" (UID: \"e194051c-28aa-4275-8a63-ce4ce433e23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:08.153915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.153904    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f84cce-9t7jr\" (UID: \"e194051c-28aa-4275-8a63-ce4ce433e23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:08.254917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.254883    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f84cce-9t7jr\" (UID: \"e194051c-28aa-4275-8a63-ce4ce433e23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:08.255111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.255022    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzmx\" (UniqueName: \"kubernetes.io/projected/e194051c-28aa-4275-8a63-ce4ce433e23d-kube-api-access-cnzmx\") pod \"maas-api-revoke-sub-f1f84cce-9t7jr\" (UID: \"e194051c-28aa-4275-8a63-ce4ce433e23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:08.255111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.255068    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f84cce-9t7jr\" (UID: \"e194051c-28aa-4275-8a63-ce4ce433e23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:08.255531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.255512    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f84cce-9t7jr\" (UID: \"e194051c-28aa-4275-8a63-ce4ce433e23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:08.257492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.257474    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f84cce-9t7jr\" (UID: \"e194051c-28aa-4275-8a63-ce4ce433e23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:08.264133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.264115    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzmx\" (UniqueName: \"kubernetes.io/projected/e194051c-28aa-4275-8a63-ce4ce433e23d-kube-api-access-cnzmx\") pod \"maas-api-revoke-sub-f1f84cce-9t7jr\" (UID: \"e194051c-28aa-4275-8a63-ce4ce433e23d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:08.378642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.378617    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:08.671944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.671910    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"]
Oct 30 21:26:08.672270 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:08.672248    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode194051c_28aa_4275_8a63_ce4ce433e23d.slice/crio-0383bc65d88ba22071c00d11077900c6d6c8fd215692fdf9df7e88d9302e2a0f WatchSource:0}: Error finding container 0383bc65d88ba22071c00d11077900c6d6c8fd215692fdf9df7e88d9302e2a0f: Status 404 returned error can't find the container with id 0383bc65d88ba22071c00d11077900c6d6c8fd215692fdf9df7e88d9302e2a0f
Oct 30 21:26:08.945190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:08.945102    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr" event={"ID":"e194051c-28aa-4275-8a63-ce4ce433e23d","Type":"ContainerStarted","Data":"0383bc65d88ba22071c00d11077900c6d6c8fd215692fdf9df7e88d9302e2a0f"}
Oct 30 21:26:09.192144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.192110    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"]
Oct 30 21:26:09.196357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.196282    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:09.204177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.204152    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"]
Oct 30 21:26:09.271443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.271407    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f3b5a81-pcmwg\" (UID: \"85370ff2-fa9e-43e0-815e-3add8f2fc416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:09.271612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.271506    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f3b5a81-pcmwg\" (UID: \"85370ff2-fa9e-43e0-815e-3add8f2fc416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:09.271612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.271566    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7sdk\" (UniqueName: \"kubernetes.io/projected/85370ff2-fa9e-43e0-815e-3add8f2fc416-kube-api-access-j7sdk\") pod \"maas-api-revoke-sub-8f3b5a81-pcmwg\" (UID: \"85370ff2-fa9e-43e0-815e-3add8f2fc416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:09.373483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.373436    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f3b5a81-pcmwg\" (UID: \"85370ff2-fa9e-43e0-815e-3add8f2fc416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:09.373684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.373504    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7sdk\" (UniqueName: \"kubernetes.io/projected/85370ff2-fa9e-43e0-815e-3add8f2fc416-kube-api-access-j7sdk\") pod \"maas-api-revoke-sub-8f3b5a81-pcmwg\" (UID: \"85370ff2-fa9e-43e0-815e-3add8f2fc416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:09.373858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.373839    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f3b5a81-pcmwg\" (UID: \"85370ff2-fa9e-43e0-815e-3add8f2fc416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:09.374838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.374811    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f3b5a81-pcmwg\" (UID: \"85370ff2-fa9e-43e0-815e-3add8f2fc416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:09.376292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.376267    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f3b5a81-pcmwg\" (UID: \"85370ff2-fa9e-43e0-815e-3add8f2fc416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:09.383690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.383665    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7sdk\" (UniqueName: \"kubernetes.io/projected/85370ff2-fa9e-43e0-815e-3add8f2fc416-kube-api-access-j7sdk\") pod \"maas-api-revoke-sub-8f3b5a81-pcmwg\" (UID: \"85370ff2-fa9e-43e0-815e-3add8f2fc416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:09.525691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.525655    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:09.863367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.863340    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"]
Oct 30 21:26:09.864448 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:09.864423    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85370ff2_fa9e_43e0_815e_3add8f2fc416.slice/crio-38561119b329f9db040d9ff7c22a92b897f2d75a692a0ce752e5218ed978bf4c WatchSource:0}: Error finding container 38561119b329f9db040d9ff7c22a92b897f2d75a692a0ce752e5218ed978bf4c: Status 404 returned error can't find the container with id 38561119b329f9db040d9ff7c22a92b897f2d75a692a0ce752e5218ed978bf4c
Oct 30 21:26:09.954150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.954118    4918 generic.go:356] "Generic (PLEG): container finished" podID="e194051c-28aa-4275-8a63-ce4ce433e23d" containerID="4b9e0c79ef86af5ba7da0c78ca5708de96065dc8e956ab50f909c83f876ff0e7" exitCode=0
Oct 30 21:26:09.954282 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.954192    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr" event={"ID":"e194051c-28aa-4275-8a63-ce4ce433e23d","Type":"ContainerDied","Data":"4b9e0c79ef86af5ba7da0c78ca5708de96065dc8e956ab50f909c83f876ff0e7"}
Oct 30 21:26:09.955585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:09.955564    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg" event={"ID":"85370ff2-fa9e-43e0-815e-3add8f2fc416","Type":"ContainerStarted","Data":"38561119b329f9db040d9ff7c22a92b897f2d75a692a0ce752e5218ed978bf4c"}
Oct 30 21:26:11.082497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:11.082459    4918 generic.go:356] "Generic (PLEG): container finished" podID="85370ff2-fa9e-43e0-815e-3add8f2fc416" containerID="d6d816fcf3521e83131feb379a250e0117a594d9c880a31c7d303856cc02862e" exitCode=0
Oct 30 21:26:11.086077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:11.086050    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg" event={"ID":"85370ff2-fa9e-43e0-815e-3add8f2fc416","Type":"ContainerDied","Data":"d6d816fcf3521e83131feb379a250e0117a594d9c880a31c7d303856cc02862e"}
Oct 30 21:26:11.310320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:11.310290    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:11.413815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:11.413729    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e194051c-28aa-4275-8a63-ce4ce433e23d-kube-api-access-cnzmx\" (UniqueName: \"kubernetes.io/projected/e194051c-28aa-4275-8a63-ce4ce433e23d-kube-api-access-cnzmx\") pod \"e194051c-28aa-4275-8a63-ce4ce433e23d\" (UID: \"e194051c-28aa-4275-8a63-ce4ce433e23d\") "
Oct 30 21:26:11.413815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:11.413806    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-xks-ca\") pod \"e194051c-28aa-4275-8a63-ce4ce433e23d\" (UID: \"e194051c-28aa-4275-8a63-ce4ce433e23d\") "
Oct 30 21:26:11.414069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:11.413850    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-openshift-ca\") pod \"e194051c-28aa-4275-8a63-ce4ce433e23d\" (UID: \"e194051c-28aa-4275-8a63-ce4ce433e23d\") "
Oct 30 21:26:11.414545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:11.414248    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-openshift-ca" pod "e194051c-28aa-4275-8a63-ce4ce433e23d" (UID: "e194051c-28aa-4275-8a63-ce4ce433e23d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:11.414545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:11.414508    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.416091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:11.416068    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e194051c-28aa-4275-8a63-ce4ce433e23d-kube-api-access-cnzmx" pod "e194051c-28aa-4275-8a63-ce4ce433e23d" (UID: "e194051c-28aa-4275-8a63-ce4ce433e23d"). InnerVolumeSpecName "kube-api-access-cnzmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:11.416185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:11.416087    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-xks-ca" pod "e194051c-28aa-4275-8a63-ce4ce433e23d" (UID: "e194051c-28aa-4275-8a63-ce4ce433e23d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:11.515713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:11.515681    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnzmx\" (UniqueName: \"kubernetes.io/projected/e194051c-28aa-4275-8a63-ce4ce433e23d-kube-api-access-cnzmx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:11.515713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:11.515710    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e194051c-28aa-4275-8a63-ce4ce433e23d-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:12.091421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.091363    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"
Oct 30 21:26:12.091913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.091435    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr" event={"ID":"e194051c-28aa-4275-8a63-ce4ce433e23d","Type":"ContainerDied","Data":"0383bc65d88ba22071c00d11077900c6d6c8fd215692fdf9df7e88d9302e2a0f"}
Oct 30 21:26:12.091913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.091476    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0383bc65d88ba22071c00d11077900c6d6c8fd215692fdf9df7e88d9302e2a0f"
Oct 30 21:26:12.322100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.322064    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:12.427263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.427177    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-xks-ca\") pod \"85370ff2-fa9e-43e0-815e-3add8f2fc416\" (UID: \"85370ff2-fa9e-43e0-815e-3add8f2fc416\") "
Oct 30 21:26:12.427488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.427291    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85370ff2-fa9e-43e0-815e-3add8f2fc416-kube-api-access-j7sdk\" (UniqueName: \"kubernetes.io/projected/85370ff2-fa9e-43e0-815e-3add8f2fc416-kube-api-access-j7sdk\") pod \"85370ff2-fa9e-43e0-815e-3add8f2fc416\" (UID: \"85370ff2-fa9e-43e0-815e-3add8f2fc416\") "
Oct 30 21:26:12.427578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.427511    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-openshift-ca\") pod \"85370ff2-fa9e-43e0-815e-3add8f2fc416\" (UID: \"85370ff2-fa9e-43e0-815e-3add8f2fc416\") "
Oct 30 21:26:12.427880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.427855    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-openshift-ca" pod "85370ff2-fa9e-43e0-815e-3add8f2fc416" (UID: "85370ff2-fa9e-43e0-815e-3add8f2fc416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:12.428089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.428038    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:12.429584 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.429558    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-xks-ca" pod "85370ff2-fa9e-43e0-815e-3add8f2fc416" (UID: "85370ff2-fa9e-43e0-815e-3add8f2fc416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:12.429677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.429620    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85370ff2-fa9e-43e0-815e-3add8f2fc416-kube-api-access-j7sdk" pod "85370ff2-fa9e-43e0-815e-3add8f2fc416" (UID: "85370ff2-fa9e-43e0-815e-3add8f2fc416"). InnerVolumeSpecName "kube-api-access-j7sdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:12.540417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.537293    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7sdk\" (UniqueName: \"kubernetes.io/projected/85370ff2-fa9e-43e0-815e-3add8f2fc416-kube-api-access-j7sdk\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:12.540417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:12.537352    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85370ff2-fa9e-43e0-815e-3add8f2fc416-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:13.132283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:13.128070    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg" event={"ID":"85370ff2-fa9e-43e0-815e-3add8f2fc416","Type":"ContainerDied","Data":"38561119b329f9db040d9ff7c22a92b897f2d75a692a0ce752e5218ed978bf4c"}
Oct 30 21:26:13.132283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:13.128134    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38561119b329f9db040d9ff7c22a92b897f2d75a692a0ce752e5218ed978bf4c"
Oct 30 21:26:13.132283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:13.128206    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"
Oct 30 21:26:16.408668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.408633    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"]
Oct 30 21:26:16.409199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.409184    4918 cpu_manager.go:414] "containerMap: removing container" podUID="85370ff2-fa9e-43e0-815e-3add8f2fc416" containerName="revoke-keys"
Oct 30 21:26:16.409258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.409202    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85370ff2-fa9e-43e0-815e-3add8f2fc416" containerName="revoke-keys"
Oct 30 21:26:16.409258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.409221    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e194051c-28aa-4275-8a63-ce4ce433e23d" containerName="revoke-keys"
Oct 30 21:26:16.409258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.409226    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e194051c-28aa-4275-8a63-ce4ce433e23d" containerName="revoke-keys"
Oct 30 21:26:16.409367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.409333    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="85370ff2-fa9e-43e0-815e-3add8f2fc416" containerName="revoke-keys"
Oct 30 21:26:16.409367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.409344    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e194051c-28aa-4275-8a63-ce4ce433e23d" containerName="revoke-keys"
Oct 30 21:26:16.412731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.412709    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:16.419110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.419089    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:16.419452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.419432    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:26:16.419590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.419489    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:16.420458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.419385    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:16.424794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.424773    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"]
Oct 30 21:26:16.507343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.507312    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9661254-b9fhh\" (UID: \"33b9e0be-eef3-4fa2-902b-99364944dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:16.507560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.507350    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9661254-b9fhh\" (UID: \"33b9e0be-eef3-4fa2-902b-99364944dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:16.507610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.507558    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qt5s\" (UniqueName: \"kubernetes.io/projected/33b9e0be-eef3-4fa2-902b-99364944dec9-kube-api-access-5qt5s\") pod \"maas-api-revoke-sub-e9661254-b9fhh\" (UID: \"33b9e0be-eef3-4fa2-902b-99364944dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:16.608815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.608775    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9661254-b9fhh\" (UID: \"33b9e0be-eef3-4fa2-902b-99364944dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:16.608815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.608813    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9661254-b9fhh\" (UID: \"33b9e0be-eef3-4fa2-902b-99364944dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:16.609107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.608909    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qt5s\" (UniqueName: \"kubernetes.io/projected/33b9e0be-eef3-4fa2-902b-99364944dec9-kube-api-access-5qt5s\") pod \"maas-api-revoke-sub-e9661254-b9fhh\" (UID: \"33b9e0be-eef3-4fa2-902b-99364944dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:16.609564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.609542    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9661254-b9fhh\" (UID: \"33b9e0be-eef3-4fa2-902b-99364944dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:16.611382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.611358    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9661254-b9fhh\" (UID: \"33b9e0be-eef3-4fa2-902b-99364944dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:16.628535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.628512    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qt5s\" (UniqueName: \"kubernetes.io/projected/33b9e0be-eef3-4fa2-902b-99364944dec9-kube-api-access-5qt5s\") pod \"maas-api-revoke-sub-e9661254-b9fhh\" (UID: \"33b9e0be-eef3-4fa2-902b-99364944dec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:16.729526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:16.729498    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:17.055807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:17.055781    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"]
Oct 30 21:26:17.056449 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:17.056418    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b9e0be_eef3_4fa2_902b_99364944dec9.slice/crio-11d4bb8885737ca73d046fec99161a7079c362676962db7130664f0a73470c67 WatchSource:0}: Error finding container 11d4bb8885737ca73d046fec99161a7079c362676962db7130664f0a73470c67: Status 404 returned error can't find the container with id 11d4bb8885737ca73d046fec99161a7079c362676962db7130664f0a73470c67
Oct 30 21:26:17.158727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:17.158698    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh" event={"ID":"33b9e0be-eef3-4fa2-902b-99364944dec9","Type":"ContainerStarted","Data":"11d4bb8885737ca73d046fec99161a7079c362676962db7130664f0a73470c67"}
Oct 30 21:26:18.169559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:18.169515    4918 generic.go:356] "Generic (PLEG): container finished" podID="33b9e0be-eef3-4fa2-902b-99364944dec9" containerID="97ba78aebdb033a1c007f4f26c9ce3b6e75b3789bb31529f6a57eeacf31d79af" exitCode=0
Oct 30 21:26:18.170052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:18.169581    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh" event={"ID":"33b9e0be-eef3-4fa2-902b-99364944dec9","Type":"ContainerDied","Data":"97ba78aebdb033a1c007f4f26c9ce3b6e75b3789bb31529f6a57eeacf31d79af"}
Oct 30 21:26:19.343180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.343153    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:19.445838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.445742    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b9e0be-eef3-4fa2-902b-99364944dec9-kube-api-access-5qt5s\" (UniqueName: \"kubernetes.io/projected/33b9e0be-eef3-4fa2-902b-99364944dec9-kube-api-access-5qt5s\") pod \"33b9e0be-eef3-4fa2-902b-99364944dec9\" (UID: \"33b9e0be-eef3-4fa2-902b-99364944dec9\") "
Oct 30 21:26:19.446022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.445864    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-xks-ca\") pod \"33b9e0be-eef3-4fa2-902b-99364944dec9\" (UID: \"33b9e0be-eef3-4fa2-902b-99364944dec9\") "
Oct 30 21:26:19.446022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.446001    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-openshift-ca\") pod \"33b9e0be-eef3-4fa2-902b-99364944dec9\" (UID: \"33b9e0be-eef3-4fa2-902b-99364944dec9\") "
Oct 30 21:26:19.446413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.446360    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-openshift-ca" pod "33b9e0be-eef3-4fa2-902b-99364944dec9" (UID: "33b9e0be-eef3-4fa2-902b-99364944dec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:19.446595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.446554    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:19.448068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.448046    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-xks-ca" pod "33b9e0be-eef3-4fa2-902b-99364944dec9" (UID: "33b9e0be-eef3-4fa2-902b-99364944dec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:19.448068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.448058    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b9e0be-eef3-4fa2-902b-99364944dec9-kube-api-access-5qt5s" pod "33b9e0be-eef3-4fa2-902b-99364944dec9" (UID: "33b9e0be-eef3-4fa2-902b-99364944dec9"). InnerVolumeSpecName "kube-api-access-5qt5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:19.525051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.525011    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"]
Oct 30 21:26:19.525670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.525651    4918 cpu_manager.go:414] "containerMap: removing container" podUID="33b9e0be-eef3-4fa2-902b-99364944dec9" containerName="revoke-keys"
Oct 30 21:26:19.525795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.525672    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b9e0be-eef3-4fa2-902b-99364944dec9" containerName="revoke-keys"
Oct 30 21:26:19.525871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.525831    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b9e0be-eef3-4fa2-902b-99364944dec9" containerName="revoke-keys"
Oct 30 21:26:19.529442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.529424    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:19.539900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.539876    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"]
Oct 30 21:26:19.557251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.557215    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f59e43a-27gk7\" (UID: \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:19.557450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.557338    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrf6\" (UniqueName: \"kubernetes.io/projected/1aed7b57-d027-49d4-92c0-7848f4cb3c78-kube-api-access-fwrf6\") pod \"maas-api-revoke-sub-6f59e43a-27gk7\" (UID: \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:19.557450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.557373    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f59e43a-27gk7\" (UID: \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:19.557619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.557457    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qt5s\" (UniqueName: \"kubernetes.io/projected/33b9e0be-eef3-4fa2-902b-99364944dec9-kube-api-access-5qt5s\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:19.557619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.557470    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b9e0be-eef3-4fa2-902b-99364944dec9-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:19.659031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.659000    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f59e43a-27gk7\" (UID: \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:19.659260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.659087    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrf6\" (UniqueName: \"kubernetes.io/projected/1aed7b57-d027-49d4-92c0-7848f4cb3c78-kube-api-access-fwrf6\") pod \"maas-api-revoke-sub-6f59e43a-27gk7\" (UID: \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:19.659260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.659115    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f59e43a-27gk7\" (UID: \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:19.659925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.659901    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f59e43a-27gk7\" (UID: \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:19.661754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.661726    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f59e43a-27gk7\" (UID: \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:19.682278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.682254    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrf6\" (UniqueName: \"kubernetes.io/projected/1aed7b57-d027-49d4-92c0-7848f4cb3c78-kube-api-access-fwrf6\") pod \"maas-api-revoke-sub-6f59e43a-27gk7\" (UID: \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:19.834253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.834221    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"]
Oct 30 21:26:19.838537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.838514    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:19.846911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.846885    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:19.850115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.850097    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"]
Oct 30 21:26:19.864267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.864242    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af46d705-q4hff\" (UID: \"ebd9d723-0315-4c6c-8c77-a82370aecac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:19.864532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.864509    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af46d705-q4hff\" (UID: \"ebd9d723-0315-4c6c-8c77-a82370aecac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:19.864612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.864594    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79zw9\" (UniqueName: \"kubernetes.io/projected/ebd9d723-0315-4c6c-8c77-a82370aecac4-kube-api-access-79zw9\") pod \"maas-api-revoke-sub-af46d705-q4hff\" (UID: \"ebd9d723-0315-4c6c-8c77-a82370aecac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:19.965931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.965901    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af46d705-q4hff\" (UID: \"ebd9d723-0315-4c6c-8c77-a82370aecac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:19.966106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.965956    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79zw9\" (UniqueName: \"kubernetes.io/projected/ebd9d723-0315-4c6c-8c77-a82370aecac4-kube-api-access-79zw9\") pod \"maas-api-revoke-sub-af46d705-q4hff\" (UID: \"ebd9d723-0315-4c6c-8c77-a82370aecac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:19.966106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.966014    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af46d705-q4hff\" (UID: \"ebd9d723-0315-4c6c-8c77-a82370aecac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:19.966694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.966671    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af46d705-q4hff\" (UID: \"ebd9d723-0315-4c6c-8c77-a82370aecac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:19.968297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.968273    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af46d705-q4hff\" (UID: \"ebd9d723-0315-4c6c-8c77-a82370aecac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:19.975760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:19.975738    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79zw9\" (UniqueName: \"kubernetes.io/projected/ebd9d723-0315-4c6c-8c77-a82370aecac4-kube-api-access-79zw9\") pod \"maas-api-revoke-sub-af46d705-q4hff\" (UID: \"ebd9d723-0315-4c6c-8c77-a82370aecac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:20.134018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.133987    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"]
Oct 30 21:26:20.134164 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:20.134109    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aed7b57_d027_49d4_92c0_7848f4cb3c78.slice/crio-a1ac49dbb5aed5512fb0ba0bfd6c2728f668229d3ebd21f4e0514743bda5f6e3 WatchSource:0}: Error finding container a1ac49dbb5aed5512fb0ba0bfd6c2728f668229d3ebd21f4e0514743bda5f6e3: Status 404 returned error can't find the container with id a1ac49dbb5aed5512fb0ba0bfd6c2728f668229d3ebd21f4e0514743bda5f6e3
Oct 30 21:26:20.155148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.155126    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:20.190411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.189691    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"
Oct 30 21:26:20.190411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.189720    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh" event={"ID":"33b9e0be-eef3-4fa2-902b-99364944dec9","Type":"ContainerDied","Data":"11d4bb8885737ca73d046fec99161a7079c362676962db7130664f0a73470c67"}
Oct 30 21:26:20.190411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.189793    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11d4bb8885737ca73d046fec99161a7079c362676962db7130664f0a73470c67"
Oct 30 21:26:20.194405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.191438    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7" event={"ID":"1aed7b57-d027-49d4-92c0-7848f4cb3c78","Type":"ContainerStarted","Data":"a1ac49dbb5aed5512fb0ba0bfd6c2728f668229d3ebd21f4e0514743bda5f6e3"}
Oct 30 21:26:20.440286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.440262    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"]
Oct 30 21:26:20.487661 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:20.487629    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd9d723_0315_4c6c_8c77_a82370aecac4.slice/crio-07428264a443331779d3730c134b41fc1918205b500f6a440297e5a6ddd82453 WatchSource:0}: Error finding container 07428264a443331779d3730c134b41fc1918205b500f6a440297e5a6ddd82453: Status 404 returned error can't find the container with id 07428264a443331779d3730c134b41fc1918205b500f6a440297e5a6ddd82453
Oct 30 21:26:20.523325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.523301    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"]
Oct 30 21:26:20.539551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.539419    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:20.539551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.539422    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"]
Oct 30 21:26:20.630384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.630349    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10a20d3-74fb2\" (UID: \"c9170164-5c52-4eee-879d-a88656e9e4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:20.630568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.630470    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10a20d3-74fb2\" (UID: \"c9170164-5c52-4eee-879d-a88656e9e4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:20.630611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.630561    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddbz8\" (UniqueName: \"kubernetes.io/projected/c9170164-5c52-4eee-879d-a88656e9e4f8-kube-api-access-ddbz8\") pod \"maas-api-revoke-sub-e10a20d3-74fb2\" (UID: \"c9170164-5c52-4eee-879d-a88656e9e4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:20.742736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.733413    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10a20d3-74fb2\" (UID: \"c9170164-5c52-4eee-879d-a88656e9e4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:20.742736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.741735    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10a20d3-74fb2\" (UID: \"c9170164-5c52-4eee-879d-a88656e9e4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:20.742736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.741857    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddbz8\" (UniqueName: \"kubernetes.io/projected/c9170164-5c52-4eee-879d-a88656e9e4f8-kube-api-access-ddbz8\") pod \"maas-api-revoke-sub-e10a20d3-74fb2\" (UID: \"c9170164-5c52-4eee-879d-a88656e9e4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:20.742736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.738271    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10a20d3-74fb2\" (UID: \"c9170164-5c52-4eee-879d-a88656e9e4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:20.743639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.743428    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10a20d3-74fb2\" (UID: \"c9170164-5c52-4eee-879d-a88656e9e4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:20.751656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.751628    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddbz8\" (UniqueName: \"kubernetes.io/projected/c9170164-5c52-4eee-879d-a88656e9e4f8-kube-api-access-ddbz8\") pod \"maas-api-revoke-sub-e10a20d3-74fb2\" (UID: \"c9170164-5c52-4eee-879d-a88656e9e4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:20.860661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:20.860631    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:21.137817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:21.137782    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"]
Oct 30 21:26:21.142553 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:21.142529    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9170164_5c52_4eee_879d_a88656e9e4f8.slice/crio-dde8351270a3995b567638fa97a21bc3c436c1ec7d73a11dd86dce845b3e75ae WatchSource:0}: Error finding container dde8351270a3995b567638fa97a21bc3c436c1ec7d73a11dd86dce845b3e75ae: Status 404 returned error can't find the container with id dde8351270a3995b567638fa97a21bc3c436c1ec7d73a11dd86dce845b3e75ae
Oct 30 21:26:21.203074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:21.203042    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2" event={"ID":"c9170164-5c52-4eee-879d-a88656e9e4f8","Type":"ContainerStarted","Data":"dde8351270a3995b567638fa97a21bc3c436c1ec7d73a11dd86dce845b3e75ae"}
Oct 30 21:26:21.204664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:21.204636    4918 generic.go:356] "Generic (PLEG): container finished" podID="1aed7b57-d027-49d4-92c0-7848f4cb3c78" containerID="92c6df04aad3935c76fa48987a898e4400d3ef98a7bc2bfe3592e8f5f6424cfd" exitCode=0
Oct 30 21:26:21.204777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:21.204757    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7" event={"ID":"1aed7b57-d027-49d4-92c0-7848f4cb3c78","Type":"ContainerDied","Data":"92c6df04aad3935c76fa48987a898e4400d3ef98a7bc2bfe3592e8f5f6424cfd"}
Oct 30 21:26:21.206349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:21.206328    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff" event={"ID":"ebd9d723-0315-4c6c-8c77-a82370aecac4","Type":"ContainerStarted","Data":"75393997f786a94d1778f8b9904c10062877e398578df9fcd49bfafc30cf3294"}
Oct 30 21:26:21.206456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:21.206374    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff" event={"ID":"ebd9d723-0315-4c6c-8c77-a82370aecac4","Type":"ContainerStarted","Data":"07428264a443331779d3730c134b41fc1918205b500f6a440297e5a6ddd82453"}
Oct 30 21:26:21.244572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:21.244529    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff" podStartSLOduration=2.244516942 podStartE2EDuration="2.244516942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:21.240855963 +0000 UTC m=+1494.287415108" watchObservedRunningTime="2026-10-30 21:26:21.244516942 +0000 UTC m=+1494.291076213"
Oct 30 21:26:22.283318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.280645    4918 generic.go:356] "Generic (PLEG): container finished" podID="ebd9d723-0315-4c6c-8c77-a82370aecac4" containerID="75393997f786a94d1778f8b9904c10062877e398578df9fcd49bfafc30cf3294" exitCode=0
Oct 30 21:26:22.283318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.280948    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff" event={"ID":"ebd9d723-0315-4c6c-8c77-a82370aecac4","Type":"ContainerDied","Data":"75393997f786a94d1778f8b9904c10062877e398578df9fcd49bfafc30cf3294"}
Oct 30 21:26:22.287078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.285316    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2" event={"ID":"c9170164-5c52-4eee-879d-a88656e9e4f8","Type":"ContainerStarted","Data":"cdaddd6af92fafe3b848b226bd7114cd83ef81249bb1642798390ce829425357"}
Oct 30 21:26:22.367412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.363508    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2" podStartSLOduration=2.363486133 podStartE2EDuration="2.363486133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:22.360296666 +0000 UTC m=+1495.406855814" watchObservedRunningTime="2026-10-30 21:26:22.363486133 +0000 UTC m=+1495.410045279"
Oct 30 21:26:22.632360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.632331    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:22.752477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.747704    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-xks-ca\") pod \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\" (UID: \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\") "
Oct 30 21:26:22.752477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.747793    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aed7b57-d027-49d4-92c0-7848f4cb3c78-kube-api-access-fwrf6\" (UniqueName: \"kubernetes.io/projected/1aed7b57-d027-49d4-92c0-7848f4cb3c78-kube-api-access-fwrf6\") pod \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\" (UID: \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\") "
Oct 30 21:26:22.752477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.747986    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-openshift-ca\") pod \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\" (UID: \"1aed7b57-d027-49d4-92c0-7848f4cb3c78\") "
Oct 30 21:26:22.769098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.769059    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-openshift-ca" pod "1aed7b57-d027-49d4-92c0-7848f4cb3c78" (UID: "1aed7b57-d027-49d4-92c0-7848f4cb3c78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:22.820434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.808618    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-xks-ca" pod "1aed7b57-d027-49d4-92c0-7848f4cb3c78" (UID: "1aed7b57-d027-49d4-92c0-7848f4cb3c78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:22.848686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.848648    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aed7b57-d027-49d4-92c0-7848f4cb3c78-kube-api-access-fwrf6" pod "1aed7b57-d027-49d4-92c0-7848f4cb3c78" (UID: "1aed7b57-d027-49d4-92c0-7848f4cb3c78"). InnerVolumeSpecName "kube-api-access-fwrf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:22.892413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.875416    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:22.908525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.908471    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aed7b57-d027-49d4-92c0-7848f4cb3c78-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:22.908849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:22.908818    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwrf6\" (UniqueName: \"kubernetes.io/projected/1aed7b57-d027-49d4-92c0-7848f4cb3c78-kube-api-access-fwrf6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:23.371152 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:23.371114    4918 generic.go:356] "Generic (PLEG): container finished" podID="c9170164-5c52-4eee-879d-a88656e9e4f8" containerID="cdaddd6af92fafe3b848b226bd7114cd83ef81249bb1642798390ce829425357" exitCode=0
Oct 30 21:26:23.387829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:23.387788    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2" event={"ID":"c9170164-5c52-4eee-879d-a88656e9e4f8","Type":"ContainerDied","Data":"cdaddd6af92fafe3b848b226bd7114cd83ef81249bb1642798390ce829425357"}
Oct 30 21:26:23.396147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:23.396119    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"
Oct 30 21:26:23.446118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:23.416838    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7" event={"ID":"1aed7b57-d027-49d4-92c0-7848f4cb3c78","Type":"ContainerDied","Data":"a1ac49dbb5aed5512fb0ba0bfd6c2728f668229d3ebd21f4e0514743bda5f6e3"}
Oct 30 21:26:23.446118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:23.416895    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1ac49dbb5aed5512fb0ba0bfd6c2728f668229d3ebd21f4e0514743bda5f6e3"
Oct 30 21:26:23.753541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:23.753493    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:23.758416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:23.755457    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:23.812590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:23.812529    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:26:23.934554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:23.934511    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:26:24.282026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.281981    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:24.337048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.282374    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:26:24.337048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.282431    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:24.337048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.283573    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:24.537014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.481890    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:26:24.633425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.632874    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:26:24.812615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.693655    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:24.812615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.715233    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:24.812615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.762689    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:24.812615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.768212    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:26:24.812615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.778873    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:24.812615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.794814    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:26:24.946634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.939311    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:24.946634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:24.940099    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:26:25.301050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.301003    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:26:25.524418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.329953    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.536416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.329989    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.536416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.365495    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.536416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.365563    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:26:25.567716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.567608    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.661101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.661054    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.727892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.670513    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Oct 30 21:26:25.728150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.675489    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.728257 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.717592    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.766580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.763047    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:26:25.766580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.766487    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.842539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.777855    4918 patch_prober.go:29] interesting pod/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:26:25.842539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.777927    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" podUID="f82566da-633b-47a2-8b06-389302d80839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.842539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.785517    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:25.933026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.932856    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:25.960932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:25.940586    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:26.009072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.009033    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd9d723-0315-4c6c-8c77-a82370aecac4-kube-api-access-79zw9\" (UniqueName: \"kubernetes.io/projected/ebd9d723-0315-4c6c-8c77-a82370aecac4-kube-api-access-79zw9\") pod \"ebd9d723-0315-4c6c-8c77-a82370aecac4\" (UID: \"ebd9d723-0315-4c6c-8c77-a82370aecac4\") "
Oct 30 21:26:26.022629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.022599    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-openshift-ca\") pod \"ebd9d723-0315-4c6c-8c77-a82370aecac4\" (UID: \"ebd9d723-0315-4c6c-8c77-a82370aecac4\") "
Oct 30 21:26:26.022992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.022967    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-xks-ca\") pod \"ebd9d723-0315-4c6c-8c77-a82370aecac4\" (UID: \"ebd9d723-0315-4c6c-8c77-a82370aecac4\") "
Oct 30 21:26:26.092777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.084789    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-xks-ca" pod "ebd9d723-0315-4c6c-8c77-a82370aecac4" (UID: "ebd9d723-0315-4c6c-8c77-a82370aecac4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:26.138435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.117267    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:26:26.371497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.346813    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:26:26.386532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.386503    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:26:26.411543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.411514    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:26:26.436416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.431144    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:26:26.436416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.431205    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:26:26.436416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.431241    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:26:26.436416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.431275    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:26:26.436416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.431306    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:26:26.436416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.431339    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:26:26.436416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.164732    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:26.436416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.368802    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:26.436416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.176477    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-openshift-ca" pod "ebd9d723-0315-4c6c-8c77-a82370aecac4" (UID: "ebd9d723-0315-4c6c-8c77-a82370aecac4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:26.436416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.353585    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd9d723-0315-4c6c-8c77-a82370aecac4-kube-api-access-79zw9" pod "ebd9d723-0315-4c6c-8c77-a82370aecac4" (UID: "ebd9d723-0315-4c6c-8c77-a82370aecac4"). InnerVolumeSpecName "kube-api-access-79zw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:26.436416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.258490    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf" podUID="83a3d45f-aa19-4b00-b461-2896fe5929f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:26.597685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.558808    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:26.598661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.598623    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:26:26.599191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.599164    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:26:26.599715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.599692    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:26:26.600211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.600188    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:26:26.611259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.611211    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79zw9\" (UniqueName: \"kubernetes.io/projected/ebd9d723-0315-4c6c-8c77-a82370aecac4-kube-api-access-79zw9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:26.634854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.614422    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebd9d723-0315-4c6c-8c77-a82370aecac4-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:26.660775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.647649    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:26:26.664796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.661160    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:26:26.664796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.661533    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:26:26.673261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.673209    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:26:26.744419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.744375    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:26:26.745412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.745373    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:26:26.745601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.745588    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:26:26.745809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.745788    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:26:26.745968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.745957    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:26:26.886334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.886260    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:26.890539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.890511    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"
Oct 30 21:26:26.890782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.890751    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2" event={"ID":"c9170164-5c52-4eee-879d-a88656e9e4f8","Type":"ContainerDied","Data":"dde8351270a3995b567638fa97a21bc3c436c1ec7d73a11dd86dce845b3e75ae"}
Oct 30 21:26:26.891547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.890808    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dde8351270a3995b567638fa97a21bc3c436c1ec7d73a11dd86dce845b3e75ae"
Oct 30 21:26:26.892435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.892417    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"
Oct 30 21:26:26.892550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.892461    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff" event={"ID":"ebd9d723-0315-4c6c-8c77-a82370aecac4","Type":"ContainerDied","Data":"07428264a443331779d3730c134b41fc1918205b500f6a440297e5a6ddd82453"}
Oct 30 21:26:26.892550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:26.892499    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07428264a443331779d3730c134b41fc1918205b500f6a440297e5a6ddd82453"
Oct 30 21:26:27.026373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:27.026343    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-xks-ca\") pod \"c9170164-5c52-4eee-879d-a88656e9e4f8\" (UID: \"c9170164-5c52-4eee-879d-a88656e9e4f8\") "
Oct 30 21:26:27.026373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:27.026407    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-openshift-ca\") pod \"c9170164-5c52-4eee-879d-a88656e9e4f8\" (UID: \"c9170164-5c52-4eee-879d-a88656e9e4f8\") "
Oct 30 21:26:27.026373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:27.026460    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9170164-5c52-4eee-879d-a88656e9e4f8-kube-api-access-ddbz8\" (UniqueName: \"kubernetes.io/projected/c9170164-5c52-4eee-879d-a88656e9e4f8-kube-api-access-ddbz8\") pod \"c9170164-5c52-4eee-879d-a88656e9e4f8\" (UID: \"c9170164-5c52-4eee-879d-a88656e9e4f8\") "
Oct 30 21:26:27.027006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:27.026977    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-openshift-ca" pod "c9170164-5c52-4eee-879d-a88656e9e4f8" (UID: "c9170164-5c52-4eee-879d-a88656e9e4f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:27.028494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:27.028460    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:27.029098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:27.029074    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-xks-ca" pod "c9170164-5c52-4eee-879d-a88656e9e4f8" (UID: "c9170164-5c52-4eee-879d-a88656e9e4f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:27.030619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:27.030591    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9170164-5c52-4eee-879d-a88656e9e4f8-kube-api-access-ddbz8" pod "c9170164-5c52-4eee-879d-a88656e9e4f8" (UID: "c9170164-5c52-4eee-879d-a88656e9e4f8"). InnerVolumeSpecName "kube-api-access-ddbz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:27.129914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:27.129880    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddbz8\" (UniqueName: \"kubernetes.io/projected/c9170164-5c52-4eee-879d-a88656e9e4f8-kube-api-access-ddbz8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:27.129914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:27.129908    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9170164-5c52-4eee-879d-a88656e9e4f8-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:29.011899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.011870    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"]
Oct 30 21:26:29.012595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.012575    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c9170164-5c52-4eee-879d-a88656e9e4f8" containerName="revoke-keys"
Oct 30 21:26:29.012703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.012596    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9170164-5c52-4eee-879d-a88656e9e4f8" containerName="revoke-keys"
Oct 30 21:26:29.012703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.012631    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ebd9d723-0315-4c6c-8c77-a82370aecac4" containerName="revoke-keys"
Oct 30 21:26:29.012703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.012638    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd9d723-0315-4c6c-8c77-a82370aecac4" containerName="revoke-keys"
Oct 30 21:26:29.012703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.012676    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1aed7b57-d027-49d4-92c0-7848f4cb3c78" containerName="revoke-keys"
Oct 30 21:26:29.012703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.012681    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aed7b57-d027-49d4-92c0-7848f4cb3c78" containerName="revoke-keys"
Oct 30 21:26:29.012897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.012809    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aed7b57-d027-49d4-92c0-7848f4cb3c78" containerName="revoke-keys"
Oct 30 21:26:29.012897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.012826    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9170164-5c52-4eee-879d-a88656e9e4f8" containerName="revoke-keys"
Oct 30 21:26:29.012897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.012836    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd9d723-0315-4c6c-8c77-a82370aecac4" containerName="revoke-keys"
Oct 30 21:26:29.016910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.016887    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.036109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.036073    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"]
Oct 30 21:26:29.077068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.077031    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-985dv\" (UniqueName: \"kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-kube-api-access-985dv\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.077234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.077076    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.077234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.077160    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b01e6-c368-463f-b629-27197fe0239e-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.077356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.077281    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.077356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.077330    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.077503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.077376    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b01e6-c368-463f-b629-27197fe0239e-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.077545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.077500    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.077545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.077533    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.077644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.077629    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.161622 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:26:29.161586    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:26:29.178484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.178459    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.178629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.178533    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-985dv\" (UniqueName: \"kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-kube-api-access-985dv\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.178629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.178552    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.178629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.178570    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b01e6-c368-463f-b629-27197fe0239e-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.178808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.178682    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.178808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.178756    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.178923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.178824    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b01e6-c368-463f-b629-27197fe0239e-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.178923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.178903    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.179044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.178931    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.179044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.178997    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-certs\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.179151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.179050    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.179272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.179252    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-credential-socket\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.179351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.179330    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-data\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.179417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.179349    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b01e6-c368-463f-b629-27197fe0239e-istiod-ca-cert\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.181052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.181025    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-envoy\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.181237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.181218    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b01e6-c368-463f-b629-27197fe0239e-istio-podinfo\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.189308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.189285    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-istio-token\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.189459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.189363    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-985dv\" (UniqueName: \"kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-kube-api-access-985dv\") pod \"e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") " pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.348338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.348261    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:29.647699 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:29.647664    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0b01e6_c368_463f_b629_27197fe0239e.slice/crio-5e1559b62fa3d1164f4cadbbee0b82e0cbfde07fc5ac7512a0100cdb4446f519 WatchSource:0}: Error finding container 5e1559b62fa3d1164f4cadbbee0b82e0cbfde07fc5ac7512a0100cdb4446f519: Status 404 returned error can't find the container with id 5e1559b62fa3d1164f4cadbbee0b82e0cbfde07fc5ac7512a0100cdb4446f519
Oct 30 21:26:29.650337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.650297    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:29.650502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.650405    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:29.650502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.650420    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"]
Oct 30 21:26:29.650502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.650453    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:29.937933 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.937842    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" event={"ID":"3b0b01e6-c368-463f-b629-27197fe0239e","Type":"ContainerStarted","Data":"22e935c97622d08fb778681189e84122b1b4c05b109960b2b422bd5b1b149c54"}
Oct 30 21:26:29.937933 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.937913    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" event={"ID":"3b0b01e6-c368-463f-b629-27197fe0239e","Type":"ContainerStarted","Data":"5e1559b62fa3d1164f4cadbbee0b82e0cbfde07fc5ac7512a0100cdb4446f519"}
Oct 30 21:26:29.989413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:29.986609    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" podStartSLOduration=1.986587204 podStartE2EDuration="1.986587204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:29.962354511 +0000 UTC m=+1503.008913680" watchObservedRunningTime="2026-10-30 21:26:29.986587204 +0000 UTC m=+1503.033146340"
Oct 30 21:26:30.102311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:30.102276    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:26:30.105907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:30.105881    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:26:30.349047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:30.349013    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:30.350839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:30.350809    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 21:26:30.350969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:30.350863    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" podUID="3b0b01e6-c368-463f-b629-27197fe0239e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 21:26:30.396414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:30.396122    4918 scope.go:122] "RemoveContainer" containerID="053606bb4dc2244c12bfb187bb0de8f09affc5d384a57aad325e11494f199d8d"
Oct 30 21:26:30.428481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:30.428456    4918 scope.go:122] "RemoveContainer" containerID="80d5602e511e90fbbb9d2ffd6b399a9cbc45b8f3243c2ebe2f37d9db4441e279"
Oct 30 21:26:31.349571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:31.349534    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 21:26:31.349954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:31.349594    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" podUID="3b0b01e6-c368-463f-b629-27197fe0239e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 21:26:32.354415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.350929    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 21:26:32.354415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.350991    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" podUID="3b0b01e6-c368-463f-b629-27197fe0239e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 21:26:32.533081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.533050    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"]
Oct 30 21:26:32.539196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.539170    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.553702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.553679    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-credential-socket\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.553814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.553714    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vq6t\" (UniqueName: \"kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-kube-api-access-5vq6t\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.553854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.553818    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-socket\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.553965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.553944    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b524a7be-3268-4672-a309-a4888c48cf87-istio-podinfo\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.554010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.553997    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b524a7be-3268-4672-a309-a4888c48cf87-istiod-ca-cert\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.554079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.554068    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-data\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.554125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.554094    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-envoy\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.554187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.554175    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-istio-token\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.554227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.554206    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-certs\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.561867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.561841    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"]
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.673302    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-certs\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.673408    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-credential-socket\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.673448    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vq6t\" (UniqueName: \"kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-kube-api-access-5vq6t\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.673520    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-socket\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.673646    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b524a7be-3268-4672-a309-a4888c48cf87-istio-podinfo\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.673672    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b524a7be-3268-4672-a309-a4888c48cf87-istiod-ca-cert\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.673777    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-data\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.673827    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-envoy\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.673897    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-istio-token\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.674737    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-certs\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.675038    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-credential-socket\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.675702    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-data\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.680462    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-socket\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.694508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.681599    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b524a7be-3268-4672-a309-a4888c48cf87-istiod-ca-cert\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.732457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.703375    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-envoy\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.732457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.716762    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-726a18-openshift-default-dockercfg-b45mt\""
Oct 30 21:26:32.740798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.740323    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-istio-token\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.809418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.804123    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b524a7be-3268-4672-a309-a4888c48cf87-istio-podinfo\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.820415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.814041    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vq6t\" (UniqueName: \"kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-kube-api-access-5vq6t\") pod \"e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") " pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:32.856445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:32.853656    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:33.248377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.248330    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:33.248565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.248446    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:33.248565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.248491    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:33.255591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.255568    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"]
Oct 30 21:26:33.349547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.349523    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 21:26:33.349661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.349569    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" podUID="3b0b01e6-c368-463f-b629-27197fe0239e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 21:26:33.561163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.561131    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"]
Oct 30 21:26:33.565329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.565296    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"
Oct 30 21:26:33.639527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.631233    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrp2d\" (UniqueName: \"kubernetes.io/projected/87183b40-b615-4b61-b9bd-a26fb3da3d7a-kube-api-access-xrp2d\") pod \"payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr\" (UID: \"87183b40-b615-4b61-b9bd-a26fb3da3d7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"
Oct 30 21:26:33.639527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.631459    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87183b40-b615-4b61-b9bd-a26fb3da3d7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr\" (UID: \"87183b40-b615-4b61-b9bd-a26fb3da3d7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"
Oct 30 21:26:33.663428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.662246    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"]
Oct 30 21:26:33.696162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.679647    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-06bb0f\""
Oct 30 21:26:33.696162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.679815    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-06bb0f-dockercfg-5l9hg\""
Oct 30 21:26:33.823541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.796350    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrp2d\" (UniqueName: \"kubernetes.io/projected/87183b40-b615-4b61-b9bd-a26fb3da3d7a-kube-api-access-xrp2d\") pod \"payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr\" (UID: \"87183b40-b615-4b61-b9bd-a26fb3da3d7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"
Oct 30 21:26:33.823541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.796779    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87183b40-b615-4b61-b9bd-a26fb3da3d7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr\" (UID: \"87183b40-b615-4b61-b9bd-a26fb3da3d7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"
Oct 30 21:26:33.823541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.798132    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87183b40-b615-4b61-b9bd-a26fb3da3d7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr\" (UID: \"87183b40-b615-4b61-b9bd-a26fb3da3d7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"
Oct 30 21:26:33.882444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.882358    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrp2d\" (UniqueName: \"kubernetes.io/projected/87183b40-b615-4b61-b9bd-a26fb3da3d7a-kube-api-access-xrp2d\") pod \"payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr\" (UID: \"87183b40-b615-4b61-b9bd-a26fb3da3d7a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"
Oct 30 21:26:33.927379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.927347    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"
Oct 30 21:26:33.980341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.980265    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" event={"ID":"b524a7be-3268-4672-a309-a4888c48cf87","Type":"ContainerStarted","Data":"95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1"}
Oct 30 21:26:33.980341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:33.980315    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" event={"ID":"b524a7be-3268-4672-a309-a4888c48cf87","Type":"ContainerStarted","Data":"fa0414fc1caf3dc3cf76aaa56b560993b2ac454ef8f269bf0a2fac3177250127"}
Oct 30 21:26:34.025138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:34.025095    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" podStartSLOduration=2.025080691 podStartE2EDuration="2.025080691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:34.019534115 +0000 UTC m=+1507.066093261" watchObservedRunningTime="2026-10-30 21:26:34.025080691 +0000 UTC m=+1507.071639836"
Oct 30 21:26:34.272478 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:34.272448    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87183b40_b615_4b61_b9bd_a26fb3da3d7a.slice/crio-3d40357d4f7c7e4a482da5e186ba6490baa8e949a8714facc76d6bd78980b6fd WatchSource:0}: Error finding container 3d40357d4f7c7e4a482da5e186ba6490baa8e949a8714facc76d6bd78980b6fd: Status 404 returned error can't find the container with id 3d40357d4f7c7e4a482da5e186ba6490baa8e949a8714facc76d6bd78980b6fd
Oct 30 21:26:34.274871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:34.274845    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"]
Oct 30 21:26:34.349514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:34.349486    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 21:26:34.349624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:34.349538    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" podUID="3b0b01e6-c368-463f-b629-27197fe0239e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 21:26:34.855249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:34.855216    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:34.862870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:34.862834    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:35.013460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:35.013423    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr" event={"ID":"87183b40-b615-4b61-b9bd-a26fb3da3d7a","Type":"ContainerStarted","Data":"4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1"}
Oct 30 21:26:35.013652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:35.013485    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr" event={"ID":"87183b40-b615-4b61-b9bd-a26fb3da3d7a","Type":"ContainerStarted","Data":"3d40357d4f7c7e4a482da5e186ba6490baa8e949a8714facc76d6bd78980b6fd"}
Oct 30 21:26:35.013652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:35.013566    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"
Oct 30 21:26:35.014446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:35.013975    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:35.014936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:35.014919    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:26:35.033421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:35.033363    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr" podStartSLOduration=2.033352802 podStartE2EDuration="2.033352802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:35.032263202 +0000 UTC m=+1508.078822349" watchObservedRunningTime="2026-10-30 21:26:35.033352802 +0000 UTC m=+1508.079911946"
Oct 30 21:26:35.348687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:35.348652    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 21:26:35.348867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:35.348703    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" podUID="3b0b01e6-c368-463f-b629-27197fe0239e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 21:26:36.041422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.040449    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"]
Oct 30 21:26:36.046874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.046269    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.057190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.057138    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"]
Oct 30 21:26:36.147160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.147120    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.147344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.147227    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.147432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.147343    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.147507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.147462    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.147575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.147522    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf2zd\" (UniqueName: \"kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-kube-api-access-mf2zd\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.147752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.147732    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.147788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.147778    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.147919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.147900    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.147966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.147949    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.248927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.248894    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf2zd\" (UniqueName: \"kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-kube-api-access-mf2zd\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.249089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.248961    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.249089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.248985    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.249231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.249134    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.249231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.249193    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.249334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.249292    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.249408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.249353    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.249478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.249402    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.249478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.249439    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.249583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.249532    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.249676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.249655    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-certs\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.249944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.249920    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-credential-socket\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.250062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.249955    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-data\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.250305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.250105    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istiod-ca-cert\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.251834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.251810    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-envoy\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.252310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.252291    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-podinfo\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.257701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.257676    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-token\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.258447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.257977    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf2zd\" (UniqueName: \"kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-kube-api-access-mf2zd\") pod \"e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") " pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.349185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.349091    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Oct 30 21:26:36.349185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.349152    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" podUID="3b0b01e6-c368-463f-b629-27197fe0239e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Oct 30 21:26:36.422833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.422797    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:36.712127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.712089    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"]
Oct 30 21:26:36.712344 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:36.712318    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed0cd50_1dc5_434f_aaa6_14cc9e0dff9e.slice/crio-e010528470ff0eeb4936c0ce039ccd8da4ea2c591c8ae7a9f0f1d56bcf5b07e0 WatchSource:0}: Error finding container e010528470ff0eeb4936c0ce039ccd8da4ea2c591c8ae7a9f0f1d56bcf5b07e0: Status 404 returned error can't find the container with id e010528470ff0eeb4936c0ce039ccd8da4ea2c591c8ae7a9f0f1d56bcf5b07e0
Oct 30 21:26:36.715740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.715697    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:36.715862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.715784    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:36.715862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:36.715814    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:37.054100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.054055    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" event={"ID":"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e","Type":"ContainerStarted","Data":"215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f"}
Oct 30 21:26:37.054652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.054129    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" event={"ID":"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e","Type":"ContainerStarted","Data":"e010528470ff0eeb4936c0ce039ccd8da4ea2c591c8ae7a9f0f1d56bcf5b07e0"}
Oct 30 21:26:37.091405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.091311    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" podStartSLOduration=2.091289587 podStartE2EDuration="2.091289587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:37.087544979 +0000 UTC m=+1510.134104121" watchObservedRunningTime="2026-10-30 21:26:37.091289587 +0000 UTC m=+1510.137848734"
Oct 30 21:26:37.425106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.423710    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:37.425937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.425519    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:26:37.425937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.425595    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" podUID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 21:26:37.827311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.827272    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:37.828763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.828735    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:37.830840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.830816    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:26:37.937672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.937643    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"]
Oct 30 21:26:37.943598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.943571    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"
Oct 30 21:26:37.946197 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.946167    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"]
Oct 30 21:26:37.950751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.950106    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-726a18\""
Oct 30 21:26:37.954763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.954743    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"]
Oct 30 21:26:37.968062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.968038    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"]
Oct 30 21:26:37.968427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.968142    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"
Oct 30 21:26:37.992020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:37.991979    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-726a18-dockercfg-vg9lh\""
Oct 30 21:26:38.084465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.080184    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxn9\" (UniqueName: \"kubernetes.io/projected/22a44e09-93c1-4e78-bc3e-52de37538ec3-kube-api-access-cqxn9\") pod \"payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5\" (UID: \"22a44e09-93c1-4e78-bc3e-52de37538ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"
Oct 30 21:26:38.084465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.080243    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a44e09-93c1-4e78-bc3e-52de37538ec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5\" (UID: \"22a44e09-93c1-4e78-bc3e-52de37538ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"
Oct 30 21:26:38.084465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.080300    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9\" (UID: \"4515aee4-3a67-43d0-9d60-7ad6524bf8d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"
Oct 30 21:26:38.084465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.080471    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqcft\" (UniqueName: \"kubernetes.io/projected/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-kube-api-access-qqcft\") pod \"payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9\" (UID: \"4515aee4-3a67-43d0-9d60-7ad6524bf8d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"
Oct 30 21:26:38.191417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.187631    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqcft\" (UniqueName: \"kubernetes.io/projected/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-kube-api-access-qqcft\") pod \"payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9\" (UID: \"4515aee4-3a67-43d0-9d60-7ad6524bf8d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"
Oct 30 21:26:38.191417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.187785    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxn9\" (UniqueName: \"kubernetes.io/projected/22a44e09-93c1-4e78-bc3e-52de37538ec3-kube-api-access-cqxn9\") pod \"payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5\" (UID: \"22a44e09-93c1-4e78-bc3e-52de37538ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"
Oct 30 21:26:38.191417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.187824    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a44e09-93c1-4e78-bc3e-52de37538ec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5\" (UID: \"22a44e09-93c1-4e78-bc3e-52de37538ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"
Oct 30 21:26:38.191417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.187890    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9\" (UID: \"4515aee4-3a67-43d0-9d60-7ad6524bf8d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"
Oct 30 21:26:38.191417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.188872    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9\" (UID: \"4515aee4-3a67-43d0-9d60-7ad6524bf8d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"
Oct 30 21:26:38.335471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.221654    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a44e09-93c1-4e78-bc3e-52de37538ec3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5\" (UID: \"22a44e09-93c1-4e78-bc3e-52de37538ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"
Oct 30 21:26:38.354642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.238423    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqcft\" (UniqueName: \"kubernetes.io/projected/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-kube-api-access-qqcft\") pod \"payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9\" (UID: \"4515aee4-3a67-43d0-9d60-7ad6524bf8d3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"
Oct 30 21:26:38.354891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.296024    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxn9\" (UniqueName: \"kubernetes.io/projected/22a44e09-93c1-4e78-bc3e-52de37538ec3-kube-api-access-cqxn9\") pod \"payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5\" (UID: \"22a44e09-93c1-4e78-bc3e-52de37538ec3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"
Oct 30 21:26:38.424014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.423980    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:26:38.424191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.424033    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" podUID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 21:26:38.571722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.571680    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"
Oct 30 21:26:38.642412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.633718    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"
Oct 30 21:26:38.969984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:38.969892    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"]
Oct 30 21:26:38.974167 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:38.974138    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a44e09_93c1_4e78_bc3e_52de37538ec3.slice/crio-45842a3d9dbb9a39a8cbfeba160c450d73688d1a51020b3a00e9ff3c852f0b2b WatchSource:0}: Error finding container 45842a3d9dbb9a39a8cbfeba160c450d73688d1a51020b3a00e9ff3c852f0b2b: Status 404 returned error can't find the container with id 45842a3d9dbb9a39a8cbfeba160c450d73688d1a51020b3a00e9ff3c852f0b2b
Oct 30 21:26:39.026844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.026788    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"]
Oct 30 21:26:39.029130 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:39.029102    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4515aee4_3a67_43d0_9d60_7ad6524bf8d3.slice/crio-9af1d3afc5075fb44b238eb26cf1b5b1df576631b0c1d9a57817342089924500 WatchSource:0}: Error finding container 9af1d3afc5075fb44b238eb26cf1b5b1df576631b0c1d9a57817342089924500: Status 404 returned error can't find the container with id 9af1d3afc5075fb44b238eb26cf1b5b1df576631b0c1d9a57817342089924500
Oct 30 21:26:39.065924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.065900    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"]
Oct 30 21:26:39.071630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.071609    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"
Oct 30 21:26:39.082318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.082180    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-15e79c\""
Oct 30 21:26:39.082460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.082383    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-15e79c-dockercfg-t5mxl\""
Oct 30 21:26:39.135956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.132733    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"]
Oct 30 21:26:39.204417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.203496    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x42k\" (UniqueName: \"kubernetes.io/projected/806c4d25-beb2-45d7-9df8-a107c39b08dd-kube-api-access-5x42k\") pod \"payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g\" (UID: \"806c4d25-beb2-45d7-9df8-a107c39b08dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"
Oct 30 21:26:39.216416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.216053    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806c4d25-beb2-45d7-9df8-a107c39b08dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g\" (UID: \"806c4d25-beb2-45d7-9df8-a107c39b08dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"
Oct 30 21:26:39.383477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.376091    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806c4d25-beb2-45d7-9df8-a107c39b08dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g\" (UID: \"806c4d25-beb2-45d7-9df8-a107c39b08dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"
Oct 30 21:26:39.383477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.376159    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x42k\" (UniqueName: \"kubernetes.io/projected/806c4d25-beb2-45d7-9df8-a107c39b08dd-kube-api-access-5x42k\") pod \"payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g\" (UID: \"806c4d25-beb2-45d7-9df8-a107c39b08dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"
Oct 30 21:26:39.383477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.378255    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806c4d25-beb2-45d7-9df8-a107c39b08dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g\" (UID: \"806c4d25-beb2-45d7-9df8-a107c39b08dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"
Oct 30 21:26:39.383477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.381254    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5" event={"ID":"22a44e09-93c1-4e78-bc3e-52de37538ec3","Type":"ContainerStarted","Data":"526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9"}
Oct 30 21:26:39.383477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.381314    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5" event={"ID":"22a44e09-93c1-4e78-bc3e-52de37538ec3","Type":"ContainerStarted","Data":"45842a3d9dbb9a39a8cbfeba160c450d73688d1a51020b3a00e9ff3c852f0b2b"}
Oct 30 21:26:39.398410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.397050    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"
Oct 30 21:26:39.441474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.430917    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:26:39.441474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.430977    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" podUID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 21:26:39.445978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.445947    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9" event={"ID":"4515aee4-3a67-43d0-9d60-7ad6524bf8d3","Type":"ContainerStarted","Data":"e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d"}
Oct 30 21:26:39.446211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.446195    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9" event={"ID":"4515aee4-3a67-43d0-9d60-7ad6524bf8d3","Type":"ContainerStarted","Data":"9af1d3afc5075fb44b238eb26cf1b5b1df576631b0c1d9a57817342089924500"}
Oct 30 21:26:39.448122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.446964    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"
Oct 30 21:26:39.449076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.449030    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5" podStartSLOduration=2.4490142710000002 podStartE2EDuration="2.449014271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:39.443750066 +0000 UTC m=+1512.490309215" watchObservedRunningTime="2026-10-30 21:26:39.449014271 +0000 UTC m=+1512.495573417"
Oct 30 21:26:39.481157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.481097    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x42k\" (UniqueName: \"kubernetes.io/projected/806c4d25-beb2-45d7-9df8-a107c39b08dd-kube-api-access-5x42k\") pod \"payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g\" (UID: \"806c4d25-beb2-45d7-9df8-a107c39b08dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"
Oct 30 21:26:39.525853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.525779    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9" podStartSLOduration=2.525763065 podStartE2EDuration="2.525763065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:39.509313147 +0000 UTC m=+1512.555872317" watchObservedRunningTime="2026-10-30 21:26:39.525763065 +0000 UTC m=+1512.572322211"
Oct 30 21:26:39.736011 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:39.735981    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"
Oct 30 21:26:40.253276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:40.253244    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"]
Oct 30 21:26:40.254786 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:40.254760    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806c4d25_beb2_45d7_9df8_a107c39b08dd.slice/crio-76db9225162a61bd01d804d3251171f5cccc5592c4bc8e46cf483748f78cbd55 WatchSource:0}: Error finding container 76db9225162a61bd01d804d3251171f5cccc5592c4bc8e46cf483748f78cbd55: Status 404 returned error can't find the container with id 76db9225162a61bd01d804d3251171f5cccc5592c4bc8e46cf483748f78cbd55
Oct 30 21:26:40.434966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:40.434918    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:26:40.435245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:40.435202    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" podUID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 21:26:40.467510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:40.467473    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g" event={"ID":"806c4d25-beb2-45d7-9df8-a107c39b08dd","Type":"ContainerStarted","Data":"76db9225162a61bd01d804d3251171f5cccc5592c4bc8e46cf483748f78cbd55"}
Oct 30 21:26:41.423854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.423822    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:26:41.424334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.423882    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" podUID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 21:26:41.464618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.464585    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"]
Oct 30 21:26:41.469032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.469013    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"
Oct 30 21:26:41.476869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.476839    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g" event={"ID":"806c4d25-beb2-45d7-9df8-a107c39b08dd","Type":"ContainerStarted","Data":"d339d335eba4d0afef0918350d984a827daf01a9ee8f888eb1095b34b6bfc8e1"}
Oct 30 21:26:41.477015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.476940    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"
Oct 30 21:26:41.511703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.510735    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-128d2f\""
Oct 30 21:26:41.519935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.519508    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"]
Oct 30 21:26:41.528951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.528923    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"]
Oct 30 21:26:41.529122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.529017    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"
Oct 30 21:26:41.533343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.533303    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"]
Oct 30 21:26:41.541566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.541546    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-128d2f-dockercfg-6zj2k\""
Oct 30 21:26:41.564655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.564632    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6glxg\" (UniqueName: \"kubernetes.io/projected/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-kube-api-access-6glxg\") pod \"payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f\" (UID: \"53f737dc-6ea7-4281-a418-7b84fcf2fa4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"
Oct 30 21:26:41.564872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.564849    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f\" (UID: \"53f737dc-6ea7-4281-a418-7b84fcf2fa4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"
Oct 30 21:26:41.564944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.564928    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsrfv\" (UniqueName: \"kubernetes.io/projected/0256d9fa-a55f-4457-baf2-b337961bcadd-kube-api-access-qsrfv\") pod \"payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6\" (UID: \"0256d9fa-a55f-4457-baf2-b337961bcadd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"
Oct 30 21:26:41.565168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.565150    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0256d9fa-a55f-4457-baf2-b337961bcadd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6\" (UID: \"0256d9fa-a55f-4457-baf2-b337961bcadd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"
Oct 30 21:26:41.618831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.618338    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g" podStartSLOduration=2.618317117 podStartE2EDuration="2.618317117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:41.613365418 +0000 UTC m=+1514.659924585" watchObservedRunningTime="2026-10-30 21:26:41.618317117 +0000 UTC m=+1514.664876254"
Oct 30 21:26:41.679478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.677932    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f\" (UID: \"53f737dc-6ea7-4281-a418-7b84fcf2fa4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"
Oct 30 21:26:41.679478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.678013    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsrfv\" (UniqueName: \"kubernetes.io/projected/0256d9fa-a55f-4457-baf2-b337961bcadd-kube-api-access-qsrfv\") pod \"payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6\" (UID: \"0256d9fa-a55f-4457-baf2-b337961bcadd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"
Oct 30 21:26:41.679478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.678135    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0256d9fa-a55f-4457-baf2-b337961bcadd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6\" (UID: \"0256d9fa-a55f-4457-baf2-b337961bcadd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"
Oct 30 21:26:41.679478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.678189    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6glxg\" (UniqueName: \"kubernetes.io/projected/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-kube-api-access-6glxg\") pod \"payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f\" (UID: \"53f737dc-6ea7-4281-a418-7b84fcf2fa4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"
Oct 30 21:26:41.679963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.679552    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f\" (UID: \"53f737dc-6ea7-4281-a418-7b84fcf2fa4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"
Oct 30 21:26:41.684431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.680311    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0256d9fa-a55f-4457-baf2-b337961bcadd-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6\" (UID: \"0256d9fa-a55f-4457-baf2-b337961bcadd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"
Oct 30 21:26:41.718610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.718540    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsrfv\" (UniqueName: \"kubernetes.io/projected/0256d9fa-a55f-4457-baf2-b337961bcadd-kube-api-access-qsrfv\") pod \"payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6\" (UID: \"0256d9fa-a55f-4457-baf2-b337961bcadd\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"
Oct 30 21:26:41.785230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.783852    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"
Oct 30 21:26:41.814413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.794707    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6glxg\" (UniqueName: \"kubernetes.io/projected/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-kube-api-access-6glxg\") pod \"payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f\" (UID: \"53f737dc-6ea7-4281-a418-7b84fcf2fa4c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"
Oct 30 21:26:41.845624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:41.845207    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"
Oct 30 21:26:42.346661 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:42.346628    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0256d9fa_a55f_4457_baf2_b337961bcadd.slice/crio-cea50ec0a6bf7a4c685af591f940bc8d7442fe85bab556d181ffcda66ee4c863 WatchSource:0}: Error finding container cea50ec0a6bf7a4c685af591f940bc8d7442fe85bab556d181ffcda66ee4c863: Status 404 returned error can't find the container with id cea50ec0a6bf7a4c685af591f940bc8d7442fe85bab556d181ffcda66ee4c863
Oct 30 21:26:42.347341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.347312    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"]
Oct 30 21:26:42.428045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.427994    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:26:42.428509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.428224    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" podUID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 21:26:42.454636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.454610    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"]
Oct 30 21:26:42.454901 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:42.454879    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f737dc_6ea7_4281_a418_7b84fcf2fa4c.slice/crio-4730653bc9794868504ec0cb39a946fcda7392ac85786889b22aa0808fc10126 WatchSource:0}: Error finding container 4730653bc9794868504ec0cb39a946fcda7392ac85786889b22aa0808fc10126: Status 404 returned error can't find the container with id 4730653bc9794868504ec0cb39a946fcda7392ac85786889b22aa0808fc10126
Oct 30 21:26:42.512319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.512229    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f" event={"ID":"53f737dc-6ea7-4281-a418-7b84fcf2fa4c","Type":"ContainerStarted","Data":"4730653bc9794868504ec0cb39a946fcda7392ac85786889b22aa0808fc10126"}
Oct 30 21:26:42.515899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.515064    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6" event={"ID":"0256d9fa-a55f-4457-baf2-b337961bcadd","Type":"ContainerStarted","Data":"74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de"}
Oct 30 21:26:42.515899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.515133    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6" event={"ID":"0256d9fa-a55f-4457-baf2-b337961bcadd","Type":"ContainerStarted","Data":"cea50ec0a6bf7a4c685af591f940bc8d7442fe85bab556d181ffcda66ee4c863"}
Oct 30 21:26:42.515899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.515366    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"
Oct 30 21:26:42.539235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.539189    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6" podStartSLOduration=1.5391755950000001 podStartE2EDuration="1.539175595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:42.538094561 +0000 UTC m=+1515.584653708" watchObservedRunningTime="2026-10-30 21:26:42.539175595 +0000 UTC m=+1515.585734739"
Oct 30 21:26:42.861084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.861053    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"]
Oct 30 21:26:42.865617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.865601    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:42.875778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.875754    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:42.875938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.875863    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:42.876007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.875940    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:42.876603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.876583    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:26:42.880487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.880460    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"]
Oct 30 21:26:42.946479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.946444    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rllzf\" (UniqueName: \"kubernetes.io/projected/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-kube-api-access-rllzf\") pod \"maas-api-revoke-sub-545654d1-qb7jp\" (UID: \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:42.946679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.946566    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-545654d1-qb7jp\" (UID: \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:42.946737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:42.946719    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-545654d1-qb7jp\" (UID: \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:43.047831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.047799    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-545654d1-qb7jp\" (UID: \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:43.047999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.047933    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rllzf\" (UniqueName: \"kubernetes.io/projected/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-kube-api-access-rllzf\") pod \"maas-api-revoke-sub-545654d1-qb7jp\" (UID: \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:43.047999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.047978    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-545654d1-qb7jp\" (UID: \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:43.048627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.048606    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-545654d1-qb7jp\" (UID: \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:43.050395 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.050365    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-545654d1-qb7jp\" (UID: \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:43.057767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.057744    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rllzf\" (UniqueName: \"kubernetes.io/projected/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-kube-api-access-rllzf\") pod \"maas-api-revoke-sub-545654d1-qb7jp\" (UID: \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:43.183665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.183583    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:43.424172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.424130    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 30 21:26:43.424355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.424192    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" podUID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 30 21:26:43.478855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.478827    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"]
Oct 30 21:26:43.479706 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:43.479684    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc91c205_89a4_4b6f_8ba4_1edb6d0b7c96.slice/crio-00ec97dc8a599a62f85953322fec110ed4f2b3e66cc63ee9f269ac66115587cf WatchSource:0}: Error finding container 00ec97dc8a599a62f85953322fec110ed4f2b3e66cc63ee9f269ac66115587cf: Status 404 returned error can't find the container with id 00ec97dc8a599a62f85953322fec110ed4f2b3e66cc63ee9f269ac66115587cf
Oct 30 21:26:43.524792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.524769    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f" event={"ID":"53f737dc-6ea7-4281-a418-7b84fcf2fa4c","Type":"ContainerStarted","Data":"81f586fa9e61d5bc360541cb79d4d14536243ab957ccc85a208d49675fed98bc"}
Oct 30 21:26:43.524892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.524854    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"
Oct 30 21:26:43.526244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.526222    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp" event={"ID":"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96","Type":"ContainerStarted","Data":"00ec97dc8a599a62f85953322fec110ed4f2b3e66cc63ee9f269ac66115587cf"}
Oct 30 21:26:43.549373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:43.549327    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f" podStartSLOduration=2.549314296 podStartE2EDuration="2.549314296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:43.546155627 +0000 UTC m=+1516.592714773" watchObservedRunningTime="2026-10-30 21:26:43.549314296 +0000 UTC m=+1516.595873469"
Oct 30 21:26:44.593416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:44.586754    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp" event={"ID":"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96","Type":"ContainerStarted","Data":"ec1006a65e7cc919823c51aefe2ed3bc0819b451a19e927537c8454ef098a0f9"}
Oct 30 21:26:45.025352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:45.025311    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:45.025971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:45.025944    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:45.045421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:45.043504    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:26:45.059903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:45.059716    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp" podStartSLOduration=3.059693474 podStartE2EDuration="3.059693474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:44.629766523 +0000 UTC m=+1517.676325694" watchObservedRunningTime="2026-10-30 21:26:45.059693474 +0000 UTC m=+1518.106252619"
Oct 30 21:26:45.645413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:45.645301    4918 generic.go:356] "Generic (PLEG): container finished" podID="fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96" containerID="ec1006a65e7cc919823c51aefe2ed3bc0819b451a19e927537c8454ef098a0f9" exitCode=0
Oct 30 21:26:45.665417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:45.646524    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp" event={"ID":"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96","Type":"ContainerDied","Data":"ec1006a65e7cc919823c51aefe2ed3bc0819b451a19e927537c8454ef098a0f9"}
Oct 30 21:26:46.041739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.041708    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"
Oct 30 21:26:46.125439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.125407    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"]
Oct 30 21:26:46.131254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.131231    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.159099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.158716    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"]
Oct 30 21:26:46.305468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.305047    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.305468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.305137    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.305468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.305173    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.305468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.305205    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.305468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.305322    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.305468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.305416    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.305468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.305470    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.305852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.305538    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.305852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.305610    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm2x4\" (UniqueName: \"kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-kube-api-access-bm2x4\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.432415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.429421    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.442129    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.442341    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2x4\" (UniqueName: \"kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-kube-api-access-bm2x4\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.442930    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.442997    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.443030    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.443070    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.444132    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istiod-ca-cert\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.450146    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.450504    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-certs\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.450679    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.450987    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-credential-socket\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.451340    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.452463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.451083    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-data\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.491159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.483908    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-envoy\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.491159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.486945    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-podinfo\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.491159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.487627    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm2x4\" (UniqueName: \"kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-kube-api-access-bm2x4\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.536419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.528938    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-token\") pod \"e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") " pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:46.776870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:46.776844    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:47.018418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.012578    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:47.187467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.180861    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-xks-ca\") pod \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\" (UID: \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\") "
Oct 30 21:26:47.187467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.180991    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-openshift-ca\") pod \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\" (UID: \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\") "
Oct 30 21:26:47.187467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.181163    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-kube-api-access-rllzf\" (UniqueName: \"kubernetes.io/projected/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-kube-api-access-rllzf\") pod \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\" (UID: \"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96\") "
Oct 30 21:26:47.229881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.229839    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-openshift-ca" pod "fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96" (UID: "fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:47.277451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.261589    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-kube-api-access-rllzf" pod "fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96" (UID: "fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96"). InnerVolumeSpecName "kube-api-access-rllzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:47.310447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.284331    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rllzf\" (UniqueName: \"kubernetes.io/projected/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-kube-api-access-rllzf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:47.310447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.284365    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:47.392613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.392525    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-xks-ca" pod "fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96" (UID: "fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:47.492178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.492140    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:47.592333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.592297    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"]
Oct 30 21:26:47.595810 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:47.594892    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7e7554_6fef_4b37_ac7d_1fce50662a0e.slice/crio-b44475e44c56a9d0d8c3452ac435182c8e376a02a411997ce10ae43f068da815 WatchSource:0}: Error finding container b44475e44c56a9d0d8c3452ac435182c8e376a02a411997ce10ae43f068da815: Status 404 returned error can't find the container with id b44475e44c56a9d0d8c3452ac435182c8e376a02a411997ce10ae43f068da815
Oct 30 21:26:47.598499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.598451    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:47.599589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.598547    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:47.599589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.598588    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:47.731109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.730692    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" event={"ID":"fa7e7554-6fef-4b37-ac7d-1fce50662a0e","Type":"ContainerStarted","Data":"b44475e44c56a9d0d8c3452ac435182c8e376a02a411997ce10ae43f068da815"}
Oct 30 21:26:47.743104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.743067    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp" event={"ID":"fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96","Type":"ContainerDied","Data":"00ec97dc8a599a62f85953322fec110ed4f2b3e66cc63ee9f269ac66115587cf"}
Oct 30 21:26:47.743291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.743134    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00ec97dc8a599a62f85953322fec110ed4f2b3e66cc63ee9f269ac66115587cf"
Oct 30 21:26:47.743291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:47.743220    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"
Oct 30 21:26:48.770800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:48.770761    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" event={"ID":"fa7e7554-6fef-4b37-ac7d-1fce50662a0e","Type":"ContainerStarted","Data":"5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411"}
Oct 30 21:26:48.779097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:48.779074    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:48.780184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:48.780161    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:26:48.780261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:48.780198    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" podUID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 21:26:48.818711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:48.816169    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" podStartSLOduration=2.816146523 podStartE2EDuration="2.816146523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:48.810878314 +0000 UTC m=+1521.857437474" watchObservedRunningTime="2026-10-30 21:26:48.816146523 +0000 UTC m=+1521.862705669"
Oct 30 21:26:49.778965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:49.778934    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:26:49.779416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:49.778990    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" podUID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 21:26:50.781468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:50.781289    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:26:50.781468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:50.781410    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" podUID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 21:26:51.479821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:51.479790    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"
Oct 30 21:26:51.480027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:51.479871    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"
Oct 30 21:26:51.788462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:51.783653    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:26:51.792414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:51.783718    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" podUID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 21:26:52.564618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:52.564586    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"
Oct 30 21:26:52.779680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:52.779646    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:26:52.779850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:52.779699    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" podUID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 21:26:53.264802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.264769    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"]
Oct 30 21:26:53.265381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.265368    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96" containerName="revoke-keys"
Oct 30 21:26:53.265475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.265382    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96" containerName="revoke-keys"
Oct 30 21:26:53.265516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.265490    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96" containerName="revoke-keys"
Oct 30 21:26:53.269607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.269590    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:53.280753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.280729    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:26:53.280753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.280750    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:26:53.280988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.280820    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:26:53.280988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.280864    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:26:53.282341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.282318    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"]
Oct 30 21:26:53.452176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.452142    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df686646-7klp5\" (UID: \"cea44253-7753-4a15-aae6-70d5d85da18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:53.452329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.452188    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvh5\" (UniqueName: \"kubernetes.io/projected/cea44253-7753-4a15-aae6-70d5d85da18b-kube-api-access-lwvh5\") pod \"maas-api-revoke-sub-df686646-7klp5\" (UID: \"cea44253-7753-4a15-aae6-70d5d85da18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:53.452329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.452266    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df686646-7klp5\" (UID: \"cea44253-7753-4a15-aae6-70d5d85da18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:53.530100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.530015    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"
Oct 30 21:26:53.553909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.553881    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df686646-7klp5\" (UID: \"cea44253-7753-4a15-aae6-70d5d85da18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:53.554078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.553922    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvh5\" (UniqueName: \"kubernetes.io/projected/cea44253-7753-4a15-aae6-70d5d85da18b-kube-api-access-lwvh5\") pod \"maas-api-revoke-sub-df686646-7klp5\" (UID: \"cea44253-7753-4a15-aae6-70d5d85da18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:53.554078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.554026    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df686646-7klp5\" (UID: \"cea44253-7753-4a15-aae6-70d5d85da18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:53.554547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.554528    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df686646-7klp5\" (UID: \"cea44253-7753-4a15-aae6-70d5d85da18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:53.556658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.556635    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df686646-7klp5\" (UID: \"cea44253-7753-4a15-aae6-70d5d85da18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:53.567875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.567852    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvh5\" (UniqueName: \"kubernetes.io/projected/cea44253-7753-4a15-aae6-70d5d85da18b-kube-api-access-lwvh5\") pod \"maas-api-revoke-sub-df686646-7klp5\" (UID: \"cea44253-7753-4a15-aae6-70d5d85da18b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:53.609331 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.609300    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:53.779703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.779662    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:26:53.779885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.779748    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" podUID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 21:26:53.962863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:53.962832    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"]
Oct 30 21:26:53.963298 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:53.963270    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea44253_7753_4a15_aae6_70d5d85da18b.slice/crio-67de9aafa6f1120046c47286e77724b0fa450f69ed2f725953b3399aafa08e31 WatchSource:0}: Error finding container 67de9aafa6f1120046c47286e77724b0fa450f69ed2f725953b3399aafa08e31: Status 404 returned error can't find the container with id 67de9aafa6f1120046c47286e77724b0fa450f69ed2f725953b3399aafa08e31
Oct 30 21:26:54.597656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:54.595071    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"
Oct 30 21:26:54.779460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:54.779367    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body=
Oct 30 21:26:54.779617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:54.779478    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" podUID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused"
Oct 30 21:26:54.891630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:54.891593    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5" event={"ID":"cea44253-7753-4a15-aae6-70d5d85da18b","Type":"ContainerStarted","Data":"b9cff1ea8970f951fca7eba75cb519eebe0b1e723103dffd9a92eb3d4960cd50"}
Oct 30 21:26:54.891771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:54.891701    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5" event={"ID":"cea44253-7753-4a15-aae6-70d5d85da18b","Type":"ContainerStarted","Data":"67de9aafa6f1120046c47286e77724b0fa450f69ed2f725953b3399aafa08e31"}
Oct 30 21:26:54.911401 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:54.911343    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5" podStartSLOduration=1.911329501 podStartE2EDuration="1.911329501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:54.909185575 +0000 UTC m=+1527.955744759" watchObservedRunningTime="2026-10-30 21:26:54.911329501 +0000 UTC m=+1527.957888646"
Oct 30 21:26:55.784928 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:55.784896    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:55.785775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:55.785746    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:55.786710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:55.786691    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:26:55.906342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:55.906302    4918 generic.go:356] "Generic (PLEG): container finished" podID="cea44253-7753-4a15-aae6-70d5d85da18b" containerID="b9cff1ea8970f951fca7eba75cb519eebe0b1e723103dffd9a92eb3d4960cd50" exitCode=0
Oct 30 21:26:55.906671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:55.906621    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5" event={"ID":"cea44253-7753-4a15-aae6-70d5d85da18b","Type":"ContainerDied","Data":"b9cff1ea8970f951fca7eba75cb519eebe0b1e723103dffd9a92eb3d4960cd50"}
Oct 30 21:26:57.069434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.069412    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:57.137473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.137441    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-xks-ca\") pod \"cea44253-7753-4a15-aae6-70d5d85da18b\" (UID: \"cea44253-7753-4a15-aae6-70d5d85da18b\") "
Oct 30 21:26:57.137677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.137633    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea44253-7753-4a15-aae6-70d5d85da18b-kube-api-access-lwvh5\" (UniqueName: \"kubernetes.io/projected/cea44253-7753-4a15-aae6-70d5d85da18b-kube-api-access-lwvh5\") pod \"cea44253-7753-4a15-aae6-70d5d85da18b\" (UID: \"cea44253-7753-4a15-aae6-70d5d85da18b\") "
Oct 30 21:26:57.137755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.137739    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-openshift-ca\") pod \"cea44253-7753-4a15-aae6-70d5d85da18b\" (UID: \"cea44253-7753-4a15-aae6-70d5d85da18b\") "
Oct 30 21:26:57.138154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.138107    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-openshift-ca" pod "cea44253-7753-4a15-aae6-70d5d85da18b" (UID: "cea44253-7753-4a15-aae6-70d5d85da18b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:26:57.138721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.138463    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:57.141003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.140980    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea44253-7753-4a15-aae6-70d5d85da18b-kube-api-access-lwvh5" pod "cea44253-7753-4a15-aae6-70d5d85da18b" (UID: "cea44253-7753-4a15-aae6-70d5d85da18b"). InnerVolumeSpecName "kube-api-access-lwvh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:26:57.143311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.143287    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-xks-ca" pod "cea44253-7753-4a15-aae6-70d5d85da18b" (UID: "cea44253-7753-4a15-aae6-70d5d85da18b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:26:57.239700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.239673    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea44253-7753-4a15-aae6-70d5d85da18b-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:57.239700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.239701    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwvh5\" (UniqueName: \"kubernetes.io/projected/cea44253-7753-4a15-aae6-70d5d85da18b-kube-api-access-lwvh5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:26:57.958255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.958223    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5" event={"ID":"cea44253-7753-4a15-aae6-70d5d85da18b","Type":"ContainerDied","Data":"67de9aafa6f1120046c47286e77724b0fa450f69ed2f725953b3399aafa08e31"}
Oct 30 21:26:57.958522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.958273    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67de9aafa6f1120046c47286e77724b0fa450f69ed2f725953b3399aafa08e31"
Oct 30 21:26:57.958522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:57.958329    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"
Oct 30 21:26:58.169135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.169108    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"]
Oct 30 21:26:58.169852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.169835    4918 cpu_manager.go:414] "containerMap: removing container" podUID="cea44253-7753-4a15-aae6-70d5d85da18b" containerName="revoke-keys"
Oct 30 21:26:58.169903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.169854    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea44253-7753-4a15-aae6-70d5d85da18b" containerName="revoke-keys"
Oct 30 21:26:58.169982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.169972    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea44253-7753-4a15-aae6-70d5d85da18b" containerName="revoke-keys"
Oct 30 21:26:58.173465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.173447    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.197054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.193366    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6ed261-openshift-default-dockercfg-6stmn\""
Oct 30 21:26:58.197054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.193961    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"]
Oct 30 21:26:58.262078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.262036    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63214d49-8778-4711-b6bb-6110dcc6b63b-istiod-ca-cert\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.262528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.262446    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-data\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.262680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.262665    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-socket\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.265352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.265327    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-envoy\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.265524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.265504    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hcv5\" (UniqueName: \"kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-kube-api-access-9hcv5\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.265664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.265649    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-credential-socket\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.265845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.265807    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-token\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.266020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.266002    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-certs\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.266188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.266174    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-podinfo\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.386218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.386178    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63214d49-8778-4711-b6bb-6110dcc6b63b-istiod-ca-cert\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.410682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.410647    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-data\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.410969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.410925    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-socket\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.411122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.411103    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-envoy\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.411203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.411136    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hcv5\" (UniqueName: \"kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-kube-api-access-9hcv5\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.411203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.411191    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-credential-socket\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.411312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.411222    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-token\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.411312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.411292    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-certs\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.411485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.411423    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-podinfo\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.434421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.424741    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-envoy\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.434998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.425468    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-credential-socket\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.435137 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.426032    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-certs\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.435230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.426509    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-podinfo\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.435308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.426759    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-data\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.435381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.401252    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63214d49-8778-4711-b6bb-6110dcc6b63b-istiod-ca-cert\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.435495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.426982    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-socket\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.478416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.472569    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hcv5\" (UniqueName: \"kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-kube-api-access-9hcv5\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.478416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.473077    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-token\") pod \"e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") " pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.497652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.496288    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:58.851984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.851954    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"]
Oct 30 21:26:58.852308 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:26:58.852279    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63214d49_8778_4711_b6bb_6110dcc6b63b.slice/crio-198d215fa54388aeaf0a03980dff8f7f3a8708eb65532a70d3d00473a22459eb WatchSource:0}: Error finding container 198d215fa54388aeaf0a03980dff8f7f3a8708eb65532a70d3d00473a22459eb: Status 404 returned error can't find the container with id 198d215fa54388aeaf0a03980dff8f7f3a8708eb65532a70d3d00473a22459eb
Oct 30 21:26:58.854997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.854967    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:58.855072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.855039    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:58.855122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.855071    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:26:58.970156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.970121    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" event={"ID":"63214d49-8778-4711-b6bb-6110dcc6b63b","Type":"ContainerStarted","Data":"4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf"}
Oct 30 21:26:58.970358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:58.970176    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" event={"ID":"63214d49-8778-4711-b6bb-6110dcc6b63b","Type":"ContainerStarted","Data":"198d215fa54388aeaf0a03980dff8f7f3a8708eb65532a70d3d00473a22459eb"}
Oct 30 21:26:59.002498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:59.002437    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" podStartSLOduration=1.00242017 podStartE2EDuration="1.00242017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:58.997243171 +0000 UTC m=+1532.043802317" watchObservedRunningTime="2026-10-30 21:26:59.00242017 +0000 UTC m=+1532.048979317"
Oct 30 21:26:59.381375 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:26:59.381324    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:26:59.500443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:59.497816    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:59.513532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:59.513490    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:59.978585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:59.978555    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:26:59.979655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:26:59.979626    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:27:02.625757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.625724    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"]
Oct 30 21:27:02.631694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.631673    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.638825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.638797    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"]
Oct 30 21:27:02.643068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.643045    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-a9230f-openshift-default-dockercfg-7qqkw\""
Oct 30 21:27:02.722462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.722381    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.722659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.722482    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.722659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.722618    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.722791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.722667    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.722855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.722821    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.722916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.722870    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.722916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.722897    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.723019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.722923    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.723019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.723007    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djvsw\" (UniqueName: \"kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-kube-api-access-djvsw\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.724170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.724142    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"]
Oct 30 21:27:02.729542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.729524    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.737890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.737818    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-904993-openshift-default-dockercfg-2cssf\""
Oct 30 21:27:02.741586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.741424    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"]
Oct 30 21:27:02.824653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.824605    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glz9q\" (UniqueName: \"kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-kube-api-access-glz9q\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.824826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.824662    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.824826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.824747    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-credential-socket\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.824938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.824847    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.824938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.824880    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.824938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.824926    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.825181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.825035    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djvsw\" (UniqueName: \"kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-kube-api-access-djvsw\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.825181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.825063    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-certs\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.825937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.825910    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.827750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.827717    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-token\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.827879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.827783    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-envoy\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.827879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.827858    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istiod-ca-cert\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.828004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.827901    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.828004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.827931    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.828004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.827934    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.828185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.828070    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-data\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.828185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.828110    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.828185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.828165    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.828538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.828515    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.828977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.828951    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.829063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.828986    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.829203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.829182    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-podinfo\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.829350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.829232    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.829350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.829242    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-socket\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.830428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.830365    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.841694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.841666    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djvsw\" (UniqueName: \"kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-kube-api-access-djvsw\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.842866 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.842825    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:02.930798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.930716    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istiod-ca-cert\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.930798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.930788    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-data\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.931015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.930861    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-podinfo\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.931015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.930900    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-socket\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.931015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.930951    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glz9q\" (UniqueName: \"kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-kube-api-access-glz9q\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.931015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.930997    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-credential-socket\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.931261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.931098    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-certs\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.931261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.931129    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-token\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.931261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.931161    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-envoy\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.931493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.931263    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-data\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.931594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.931570    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-credential-socket\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.931663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.931577    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istiod-ca-cert\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.931663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.931611    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-certs\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.931751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.931661    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-socket\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.933633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.933614    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-podinfo\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.933735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.933629    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-envoy\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.944125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.944104    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glz9q\" (UniqueName: \"kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-kube-api-access-glz9q\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.944314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.944293    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-token\") pod \"e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") " pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:02.951330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:02.951307    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:03.056371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:03.055719    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:03.356597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:03.355533    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"]
Oct 30 21:27:03.367621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:03.367267    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:03.367768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:03.367647    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:03.367768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:03.367716    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:03.639612 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:27:03.639584    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7e7720_3b01_4e3b_bdfe_5c771d4346aa.slice/crio-37d9eeed0dba582d03916eb44b0fdd0e82bed8bd8067636a435dfc3359b34786 WatchSource:0}: Error finding container 37d9eeed0dba582d03916eb44b0fdd0e82bed8bd8067636a435dfc3359b34786: Status 404 returned error can't find the container with id 37d9eeed0dba582d03916eb44b0fdd0e82bed8bd8067636a435dfc3359b34786
Oct 30 21:27:03.640886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:03.640863    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"]
Oct 30 21:27:03.643087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:03.643053    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:03.643188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:03.643128    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:03.643188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:03.643171    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:04.019263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:04.019230    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" event={"ID":"8ee5b7c3-c5f9-4b08-83e5-026d8df893da","Type":"ContainerStarted","Data":"f435fd66414b4c4f7b2ebc73d6407d380591d41f1bb1eca3fa5bde0287efb7c8"}
Oct 30 21:27:04.019523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:04.019284    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" event={"ID":"8ee5b7c3-c5f9-4b08-83e5-026d8df893da","Type":"ContainerStarted","Data":"6271609ef034046f0242c2d022e55510a82855ce778a9a5deb98b5d234724ad2"}
Oct 30 21:27:04.021907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:04.021872    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" event={"ID":"da7e7720-3b01-4e3b-bdfe-5c771d4346aa","Type":"ContainerStarted","Data":"c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085"}
Oct 30 21:27:04.022049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:04.021930    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" event={"ID":"da7e7720-3b01-4e3b-bdfe-5c771d4346aa","Type":"ContainerStarted","Data":"37d9eeed0dba582d03916eb44b0fdd0e82bed8bd8067636a435dfc3359b34786"}
Oct 30 21:27:04.046201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:04.046148    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" podStartSLOduration=2.046133002 podStartE2EDuration="2.046133002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:04.045380148 +0000 UTC m=+1537.091939294" watchObservedRunningTime="2026-10-30 21:27:04.046133002 +0000 UTC m=+1537.092692147"
Oct 30 21:27:04.071462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:04.071413    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" podStartSLOduration=2.071381326 podStartE2EDuration="2.071381326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:04.07060638 +0000 UTC m=+1537.117165547" watchObservedRunningTime="2026-10-30 21:27:04.071381326 +0000 UTC m=+1537.117940470"
Oct 30 21:27:04.072696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:04.072671    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:04.074628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:04.074606    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 21:27:04.074729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:04.074652    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 21:27:04.963416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:04.958831    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:04.987416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:04.985468    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:05.050893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:05.050859    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:05.051831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:05.051807    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:05.078141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:05.078121    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:06.058653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:06.058612    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:06.059364 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:06.059347    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:08.029095 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.029065    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"]
Oct 30 21:27:08.034836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.034816    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"
Oct 30 21:27:08.049709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.049687    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-904993\""
Oct 30 21:27:08.052551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.052531    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"]
Oct 30 21:27:08.056830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.056809    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"
Oct 30 21:27:08.062714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.062693    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"]
Oct 30 21:27:08.089472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.068345    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6ds2\" (UniqueName: \"kubernetes.io/projected/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-kube-api-access-g6ds2\") pod \"payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg\" (UID: \"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"
Oct 30 21:27:08.089472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.069847    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg\" (UID: \"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"
Oct 30 21:27:08.089472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.070297    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-904993-dockercfg-8lvb4\""
Oct 30 21:27:08.102942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.102898    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"]
Oct 30 21:27:08.171461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.171382    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bnsp\" (UniqueName: \"kubernetes.io/projected/893a5d07-10fd-4b2f-a4a0-db937338f5db-kube-api-access-7bnsp\") pod \"payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd\" (UID: \"893a5d07-10fd-4b2f-a4a0-db937338f5db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"
Oct 30 21:27:08.171669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.171643    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg\" (UID: \"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"
Oct 30 21:27:08.171815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.171796    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893a5d07-10fd-4b2f-a4a0-db937338f5db-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd\" (UID: \"893a5d07-10fd-4b2f-a4a0-db937338f5db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"
Oct 30 21:27:08.171983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.171961    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6ds2\" (UniqueName: \"kubernetes.io/projected/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-kube-api-access-g6ds2\") pod \"payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg\" (UID: \"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"
Oct 30 21:27:08.172428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.172374    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg\" (UID: \"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"
Oct 30 21:27:08.192377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.192346    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6ds2\" (UniqueName: \"kubernetes.io/projected/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-kube-api-access-g6ds2\") pod \"payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg\" (UID: \"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"
Oct 30 21:27:08.275427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.274340    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bnsp\" (UniqueName: \"kubernetes.io/projected/893a5d07-10fd-4b2f-a4a0-db937338f5db-kube-api-access-7bnsp\") pod \"payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd\" (UID: \"893a5d07-10fd-4b2f-a4a0-db937338f5db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"
Oct 30 21:27:08.275427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.274650    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893a5d07-10fd-4b2f-a4a0-db937338f5db-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd\" (UID: \"893a5d07-10fd-4b2f-a4a0-db937338f5db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"
Oct 30 21:27:08.275878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.275804    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893a5d07-10fd-4b2f-a4a0-db937338f5db-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd\" (UID: \"893a5d07-10fd-4b2f-a4a0-db937338f5db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"
Oct 30 21:27:08.314427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.314328    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bnsp\" (UniqueName: \"kubernetes.io/projected/893a5d07-10fd-4b2f-a4a0-db937338f5db-kube-api-access-7bnsp\") pod \"payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd\" (UID: \"893a5d07-10fd-4b2f-a4a0-db937338f5db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"
Oct 30 21:27:08.353258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.353229    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"
Oct 30 21:27:08.428699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.428661    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"
Oct 30 21:27:08.788621 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:27:08.788594    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7098ea_8a7e_4286_89e2_c3f6bf14d92b.slice/crio-aac92a4a84354805c16b1e1d93b6b298b932eda2f2feb5a3e5e2dd18b43cfc1f WatchSource:0}: Error finding container aac92a4a84354805c16b1e1d93b6b298b932eda2f2feb5a3e5e2dd18b43cfc1f: Status 404 returned error can't find the container with id aac92a4a84354805c16b1e1d93b6b298b932eda2f2feb5a3e5e2dd18b43cfc1f
Oct 30 21:27:08.789697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.789672    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"]
Oct 30 21:27:08.833291 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:27:08.833264    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893a5d07_10fd_4b2f_a4a0_db937338f5db.slice/crio-3487086a967f5e36dd4e125cb05bd86b538082819423a62450c1f6083229aa5c WatchSource:0}: Error finding container 3487086a967f5e36dd4e125cb05bd86b538082819423a62450c1f6083229aa5c: Status 404 returned error can't find the container with id 3487086a967f5e36dd4e125cb05bd86b538082819423a62450c1f6083229aa5c
Oct 30 21:27:08.834696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:08.834673    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"]
Oct 30 21:27:09.147685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:09.147211    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg" event={"ID":"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b","Type":"ContainerStarted","Data":"76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578"}
Oct 30 21:27:09.147685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:09.147272    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg" event={"ID":"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b","Type":"ContainerStarted","Data":"aac92a4a84354805c16b1e1d93b6b298b932eda2f2feb5a3e5e2dd18b43cfc1f"}
Oct 30 21:27:09.148953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:09.148931    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"
Oct 30 21:27:09.150853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:09.150830    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd" event={"ID":"893a5d07-10fd-4b2f-a4a0-db937338f5db","Type":"ContainerStarted","Data":"3487086a967f5e36dd4e125cb05bd86b538082819423a62450c1f6083229aa5c"}
Oct 30 21:27:09.185460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:09.182272    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg" podStartSLOduration=1.182259225 podStartE2EDuration="1.182259225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:09.181724504 +0000 UTC m=+1542.228283648" watchObservedRunningTime="2026-10-30 21:27:09.182259225 +0000 UTC m=+1542.228818370"
Oct 30 21:27:10.166384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:10.166342    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd" event={"ID":"893a5d07-10fd-4b2f-a4a0-db937338f5db","Type":"ContainerStarted","Data":"ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136"}
Oct 30 21:27:10.167024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:10.167001    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"
Oct 30 21:27:13.092416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.085539    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd" podStartSLOduration=5.085521065 podStartE2EDuration="5.085521065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:10.231758264 +0000 UTC m=+1543.278317432" watchObservedRunningTime="2026-10-30 21:27:13.085521065 +0000 UTC m=+1546.132080213"
Oct 30 21:27:13.115267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.101460    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"]
Oct 30 21:27:13.115267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.113949    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"
Oct 30 21:27:13.119568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.119538    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"]
Oct 30 21:27:13.125462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.125436    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-d4a0a5\""
Oct 30 21:27:13.143895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.143862    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bf3460-e153-45ef-86db-837af9418a7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6\" (UID: \"f5bf3460-e153-45ef-86db-837af9418a7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"
Oct 30 21:27:13.144026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.143972    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf26b\" (UniqueName: \"kubernetes.io/projected/f5bf3460-e153-45ef-86db-837af9418a7d-kube-api-access-kf26b\") pod \"payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6\" (UID: \"f5bf3460-e153-45ef-86db-837af9418a7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"
Oct 30 21:27:13.293416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.273685    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bf3460-e153-45ef-86db-837af9418a7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6\" (UID: \"f5bf3460-e153-45ef-86db-837af9418a7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"
Oct 30 21:27:13.293416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.273775    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf26b\" (UniqueName: \"kubernetes.io/projected/f5bf3460-e153-45ef-86db-837af9418a7d-kube-api-access-kf26b\") pod \"payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6\" (UID: \"f5bf3460-e153-45ef-86db-837af9418a7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"
Oct 30 21:27:13.293416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.275093    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bf3460-e153-45ef-86db-837af9418a7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6\" (UID: \"f5bf3460-e153-45ef-86db-837af9418a7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"
Oct 30 21:27:13.407622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.385033    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"]
Oct 30 21:27:13.407622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.397646    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf26b\" (UniqueName: \"kubernetes.io/projected/f5bf3460-e153-45ef-86db-837af9418a7d-kube-api-access-kf26b\") pod \"payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6\" (UID: \"f5bf3460-e153-45ef-86db-837af9418a7d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"
Oct 30 21:27:13.456826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.456784    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"
Oct 30 21:27:13.457286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.457262    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"]
Oct 30 21:27:13.457447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.457430    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"
Oct 30 21:27:13.476420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.467756    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-d4a0a5-dockercfg-54xql\""
Oct 30 21:27:13.571002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.570968    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd\" (UID: \"2c8d84c6-9d80-432e-bdc4-8e3684a473f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"
Oct 30 21:27:13.571205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.571095    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p4vl\" (UniqueName: \"kubernetes.io/projected/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-kube-api-access-5p4vl\") pod \"payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd\" (UID: \"2c8d84c6-9d80-432e-bdc4-8e3684a473f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"
Oct 30 21:27:13.684420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.674575    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd\" (UID: \"2c8d84c6-9d80-432e-bdc4-8e3684a473f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"
Oct 30 21:27:13.684420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.674722    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4vl\" (UniqueName: \"kubernetes.io/projected/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-kube-api-access-5p4vl\") pod \"payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd\" (UID: \"2c8d84c6-9d80-432e-bdc4-8e3684a473f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"
Oct 30 21:27:13.684420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.675869    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd\" (UID: \"2c8d84c6-9d80-432e-bdc4-8e3684a473f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"
Oct 30 21:27:13.715592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.713116    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p4vl\" (UniqueName: \"kubernetes.io/projected/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-kube-api-access-5p4vl\") pod \"payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd\" (UID: \"2c8d84c6-9d80-432e-bdc4-8e3684a473f8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"
Oct 30 21:27:13.869372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.866670    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"
Oct 30 21:27:13.923753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:13.923724    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"]
Oct 30 21:27:13.928058 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:27:13.927950    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bf3460_e153_45ef_86db_837af9418a7d.slice/crio-ed529b5de69c70badc9372708cb10ac033f86673e470bc3d546a930b98ea1c9f WatchSource:0}: Error finding container ed529b5de69c70badc9372708cb10ac033f86673e470bc3d546a930b98ea1c9f: Status 404 returned error can't find the container with id ed529b5de69c70badc9372708cb10ac033f86673e470bc3d546a930b98ea1c9f
Oct 30 21:27:14.242980 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:27:14.242953    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8d84c6_9d80_432e_bdc4_8e3684a473f8.slice/crio-8d148dc50c0a99306e2cf507e8afd8d399e034b884508f786de5d1b54ddd889a WatchSource:0}: Error finding container 8d148dc50c0a99306e2cf507e8afd8d399e034b884508f786de5d1b54ddd889a: Status 404 returned error can't find the container with id 8d148dc50c0a99306e2cf507e8afd8d399e034b884508f786de5d1b54ddd889a
Oct 30 21:27:14.243735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:14.243713    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"]
Oct 30 21:27:14.333453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:14.333420    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd" event={"ID":"2c8d84c6-9d80-432e-bdc4-8e3684a473f8","Type":"ContainerStarted","Data":"8d148dc50c0a99306e2cf507e8afd8d399e034b884508f786de5d1b54ddd889a"}
Oct 30 21:27:14.335217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:14.335193    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6" event={"ID":"f5bf3460-e153-45ef-86db-837af9418a7d","Type":"ContainerStarted","Data":"e0218e50b8a4cca9d8f18159be80ff6c201ff7549169c1242a2abf029108972d"}
Oct 30 21:27:14.335338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:14.335235    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6" event={"ID":"f5bf3460-e153-45ef-86db-837af9418a7d","Type":"ContainerStarted","Data":"ed529b5de69c70badc9372708cb10ac033f86673e470bc3d546a930b98ea1c9f"}
Oct 30 21:27:14.335434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:14.335419    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"
Oct 30 21:27:14.366875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:14.366831    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6" podStartSLOduration=1.366817536 podStartE2EDuration="1.366817536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:14.364136578 +0000 UTC m=+1547.410695728" watchObservedRunningTime="2026-10-30 21:27:14.366817536 +0000 UTC m=+1547.413376680"
Oct 30 21:27:15.345265 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:15.345230    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd" event={"ID":"2c8d84c6-9d80-432e-bdc4-8e3684a473f8","Type":"ContainerStarted","Data":"278df32917c4ad613c26799354b6f5f77982a37833ed555fd21023f3391b253f"}
Oct 30 21:27:15.345712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:15.345317    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"
Oct 30 21:27:15.379603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:15.379555    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd" podStartSLOduration=2.379541934 podStartE2EDuration="2.379541934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:15.373772078 +0000 UTC m=+1548.420331244" watchObservedRunningTime="2026-10-30 21:27:15.379541934 +0000 UTC m=+1548.426101078"
Oct 30 21:27:20.088682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:20.088102    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": read tcp 10.132.0.2:60052->10.132.0.228:15021: read: connection reset by peer" start-of-body=
Oct 30 21:27:20.088682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:20.088177    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": read tcp 10.132.0.2:60052->10.132.0.228:15021: read: connection reset by peer"
Oct 30 21:27:20.169605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:20.169583    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"
Oct 30 21:27:20.401418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:20.401312    4918 generic.go:356] "Generic (PLEG): container finished" podID="b524a7be-3268-4672-a309-a4888c48cf87" containerID="95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1" exitCode=0
Oct 30 21:27:20.401572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:20.401381    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" event={"ID":"b524a7be-3268-4672-a309-a4888c48cf87","Type":"ContainerDied","Data":"95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1"}
Oct 30 21:27:20.401918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:20.401899    4918 scope.go:122] "RemoveContainer" containerID="95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1"
Oct 30 21:27:20.403570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:20.403547    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:20.403656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:20.403612    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:20.403656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:20.403644    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:21.419108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:21.419076    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" event={"ID":"b524a7be-3268-4672-a309-a4888c48cf87","Type":"ContainerStarted","Data":"f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6"}
Oct 30 21:27:21.854548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:21.854515    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:27:21.854749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:21.854684    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:27:21.854836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:21.854739    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 21:27:22.220262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:22.220171    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"
Oct 30 21:27:22.855087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:22.855051    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:27:22.855551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:22.855103    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 21:27:23.857418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:23.857365    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:27:23.858199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:23.858163    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 21:27:24.862416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:24.861789    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:27:24.862416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:24.861875    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 21:27:25.349013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:25.348988    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"
Oct 30 21:27:25.928415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:25.901718    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body=
Oct 30 21:27:25.928415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:25.901780    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused"
Oct 30 21:27:26.357811 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:26.357771    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"
Oct 30 21:27:26.898794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:26.898678    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:27:26.899628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:26.899597    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:27:26.900626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:26.900606    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:27:29.163176 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:27:29.163140    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:27:30.488077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:30.488038    4918 scope.go:122] "RemoveContainer" containerID="72f468a25db88c736d167464dd8913dfcbcff3427d07e2cc28ec8008f8f6ca02"
Oct 30 21:27:30.499856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:30.499836    4918 scope.go:122] "RemoveContainer" containerID="cdb8146b1eea4c0576c4a129bcccff8eb82c7f75fb2547a04083348a4f776262"
Oct 30 21:27:46.021570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:46.021523    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:27:46.047623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:46.021608    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:47.434703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.434669    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"]
Oct 30 21:27:47.446806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.446783    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.468974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.468942    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-27804d-openshift-default-dockercfg-zjgbk\""
Oct 30 21:27:47.570670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.532908    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-socket\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.570670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.533033    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-credential-socket\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.570670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.533095    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-podinfo\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.570670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.533188    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-certs\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.570670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.533289    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-envoy\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.570670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.533489    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jcr\" (UniqueName: \"kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-kube-api-access-d7jcr\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.570670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.533671    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb115eb-85bb-411f-b199-707dc9df13ba-istiod-ca-cert\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.570670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.533730    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-data\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.570670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.533765    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-token\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.625426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.625369    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"]
Oct 30 21:27:47.672982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.672946    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-envoy\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.673234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.673079    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jcr\" (UniqueName: \"kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-kube-api-access-d7jcr\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.673234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.673204    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb115eb-85bb-411f-b199-707dc9df13ba-istiod-ca-cert\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.673368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.673252    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-data\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.673368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.673276    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-token\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.673368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.673346    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-socket\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.673674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.673422    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-credential-socket\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.673674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.673466    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-podinfo\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.673674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.673534    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-certs\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.674971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.674948    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-certs\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.675465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.675443    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-socket\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.675755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.675735    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-credential-socket\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.675860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.675840    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb115eb-85bb-411f-b199-707dc9df13ba-istiod-ca-cert\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.676101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.676075    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-data\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.679682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.679660    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-podinfo\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.687412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.687312    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-envoy\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.692460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.692439    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jcr\" (UniqueName: \"kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-kube-api-access-d7jcr\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.697073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.697052    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-token\") pod \"e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") " pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.804547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.776507    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:47.939283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.939206    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"]
Oct 30 21:27:47.943904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.943881    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:47.969076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:47.969051    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"]
Oct 30 21:27:48.126588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.118335    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.126588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.118420    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.126588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.118500    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.126588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.118598    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.126588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.118627    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.126588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.118688    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.126588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.118780    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.126588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.118927    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zjv\" (UniqueName: \"kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-kube-api-access-j4zjv\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.126588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.118975    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.133516 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:27:48.133480    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb115eb_85bb_411f_b199_707dc9df13ba.slice/crio-ee67b4d220fa9340ceb68a0875f68e1e1eb185013370ed7b3dc7e18bcc0b80d1 WatchSource:0}: Error finding container ee67b4d220fa9340ceb68a0875f68e1e1eb185013370ed7b3dc7e18bcc0b80d1: Status 404 returned error can't find the container with id ee67b4d220fa9340ceb68a0875f68e1e1eb185013370ed7b3dc7e18bcc0b80d1
Oct 30 21:27:48.134860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.134831    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"]
Oct 30 21:27:48.136051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.136018    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.136133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.136086    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.136133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.136119    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.220470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.220439    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.220659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.220609    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zjv\" (UniqueName: \"kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-kube-api-access-j4zjv\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.220785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.220655    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.220785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.220685    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.220785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.220716    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.221092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.220785    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.221092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.220883    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.221092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.220914    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.221092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.220990    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.221092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.220991    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.221802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.221133    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.221802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.220879    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.221802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.221476    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.222532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.222482    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.223550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.223528    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.224587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.224567    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.238593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.238569    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.238740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.238723    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zjv\" (UniqueName: \"kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-kube-api-access-j4zjv\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.262178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.262157    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:48.503492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.503446    4918 generic.go:356] "Generic (PLEG): container finished" podID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerID="f435fd66414b4c4f7b2ebc73d6407d380591d41f1bb1eca3fa5bde0287efb7c8" exitCode=0
Oct 30 21:27:48.505204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.504796    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" event={"ID":"8ee5b7c3-c5f9-4b08-83e5-026d8df893da","Type":"ContainerDied","Data":"f435fd66414b4c4f7b2ebc73d6407d380591d41f1bb1eca3fa5bde0287efb7c8"}
Oct 30 21:27:48.507127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.506766    4918 scope.go:122] "RemoveContainer" containerID="f435fd66414b4c4f7b2ebc73d6407d380591d41f1bb1eca3fa5bde0287efb7c8"
Oct 30 21:27:48.513221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.509191    4918 generic.go:356] "Generic (PLEG): container finished" podID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerID="4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf" exitCode=0
Oct 30 21:27:48.513221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.509220    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" event={"ID":"63214d49-8778-4711-b6bb-6110dcc6b63b","Type":"ContainerDied","Data":"4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf"}
Oct 30 21:27:48.513221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.512263    4918 scope.go:122] "RemoveContainer" containerID="4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf"
Oct 30 21:27:48.515269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.514847    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.515269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.514915    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.515269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.514949    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.516503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.516461    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.516695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.516667    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.516839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.516818    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.519923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.518579    4918 generic.go:356] "Generic (PLEG): container finished" podID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerID="c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085" exitCode=0
Oct 30 21:27:48.519923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.518679    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" event={"ID":"da7e7720-3b01-4e3b-bdfe-5c771d4346aa","Type":"ContainerDied","Data":"c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085"}
Oct 30 21:27:48.519923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.519084    4918 scope.go:122] "RemoveContainer" containerID="c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085"
Oct 30 21:27:48.523110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.522319    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.523110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.522420    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.523110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.522470    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.524735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.524019    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" event={"ID":"1eb115eb-85bb-411f-b199-707dc9df13ba","Type":"ContainerStarted","Data":"cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796"}
Oct 30 21:27:48.524735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.524066    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" event={"ID":"1eb115eb-85bb-411f-b199-707dc9df13ba","Type":"ContainerStarted","Data":"ee67b4d220fa9340ceb68a0875f68e1e1eb185013370ed7b3dc7e18bcc0b80d1"}
Oct 30 21:27:48.610770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.610744    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"]
Oct 30 21:27:48.612403 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.612343    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" podStartSLOduration=1.612325427 podStartE2EDuration="1.612325427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:48.598431109 +0000 UTC m=+1581.644990278" watchObservedRunningTime="2026-10-30 21:27:48.612325427 +0000 UTC m=+1581.658884575"
Oct 30 21:27:48.616912 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:27:48.616858    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5bd77e8_a7c2_4bc2_b303_bf3ea8982c79.slice/crio-c8481ee9bdd9b4c39c87a181f56b265c342c9213f6dad0b6a5c5b10ac56d31dd WatchSource:0}: Error finding container c8481ee9bdd9b4c39c87a181f56b265c342c9213f6dad0b6a5c5b10ac56d31dd: Status 404 returned error can't find the container with id c8481ee9bdd9b4c39c87a181f56b265c342c9213f6dad0b6a5c5b10ac56d31dd
Oct 30 21:27:48.630124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.630091    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.630250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.630174    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.630250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.630219    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:27:48.812056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:48.805469    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:49.603649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:49.603615    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" event={"ID":"8ee5b7c3-c5f9-4b08-83e5-026d8df893da","Type":"ContainerStarted","Data":"3a8948db74acbcb88736c86faaebf1a8c5958ac4bf6d78fc3084dadcf63edd56"}
Oct 30 21:27:49.618573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:49.607981    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" event={"ID":"63214d49-8778-4711-b6bb-6110dcc6b63b","Type":"ContainerStarted","Data":"3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c"}
Oct 30 21:27:49.669106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:49.669068    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" event={"ID":"da7e7720-3b01-4e3b-bdfe-5c771d4346aa","Type":"ContainerStarted","Data":"09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0"}
Oct 30 21:27:49.758499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:49.755717    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" event={"ID":"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79","Type":"ContainerStarted","Data":"0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78"}
Oct 30 21:27:49.758499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:49.755781    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" event={"ID":"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79","Type":"ContainerStarted","Data":"c8481ee9bdd9b4c39c87a181f56b265c342c9213f6dad0b6a5c5b10ac56d31dd"}
Oct 30 21:27:49.899948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:49.898929    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:49.939418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:49.925972    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" podStartSLOduration=2.925950013 podStartE2EDuration="2.925950013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:49.899459549 +0000 UTC m=+1582.946018713" watchObservedRunningTime="2026-10-30 21:27:49.925950013 +0000 UTC m=+1582.972509161"
Oct 30 21:27:49.953382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:49.953347    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:49.960421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:49.956505    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 21:27:49.960421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:49.956561    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 21:27:50.056528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.051045    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:50.099437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.099383    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:50.100520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.100489    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 21:27:50.100733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.100708    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 21:27:50.201634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.201541    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"]
Oct 30 21:27:50.262783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.262747    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:50.288419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.265035    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:27:50.288767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.288725    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 21:27:50.392416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.384803    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed89f361-2vc9z"]
Oct 30 21:27:50.514647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.514601    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:27:50.515322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.515297    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 21:27:50.515586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.515547    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 21:27:50.519569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.519473    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:50.610897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.610853    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:50.613366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.611505    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:50.643543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.643489    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:50.653975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.653505    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:50.678547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.678494    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:50.841560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.828530    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:50.866516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.845306    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:50.866516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.845708    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:27:50.889424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.877677    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:50.894411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.889951    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:27:50.955525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.955478    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:50.978712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.956131    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 21:27:50.981092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:50.981058    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 21:27:51.128359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.128276    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:51.130830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.130672    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 30 21:27:51.134061 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.134036    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 21:27:51.134233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.134210    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 21:27:51.169970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.169167    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:51.265368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.264859    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:27:51.265368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.264926    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 21:27:51.504183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.504144    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:51.504686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.504655    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4749573e-75c5-4d09-b8fd-4800afd463eb" path="/var/lib/kubelet/pods/4749573e-75c5-4d09-b8fd-4800afd463eb/volumes"
Oct 30 21:27:51.508739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.505505    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 21:27:51.508739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.505550    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 21:27:51.529690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.528655    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:51.614539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.614485    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:51.615930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.615905    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:27:51.662812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.662711    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:51.663560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.663538    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:27:51.696537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.696494    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:51.697358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.697333    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:27:51.857472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.847553    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:27:51.857472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.848248    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:27:51.857781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.857751    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:51.881653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.864049    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:51.882527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.882503    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:27:51.896958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.882736    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:27:51.905539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.864942    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:27:51.958510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.955995    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 21:27:51.958510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:51.956052    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 21:27:52.012451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.000487    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:27:52.012451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.000913    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:27:52.081855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.080657    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 21:27:52.081855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.080713    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 21:27:52.170633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.170541    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:52.212416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.177101    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:27:52.309599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.309562    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:27:52.413209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.413166    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 21:27:52.534542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.534509    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 21:27:52.535780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.535750    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 21:27:52.536270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.535693    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:52.576001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.575945    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:27:52.616214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.616169    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:52.629885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.629843    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:52.670512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.670466    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:52.751293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.751252    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:52.848587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.848489    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:52.894413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.893380    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:27:52.894590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.894533    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:27:52.905448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:52.898581    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:27:53.039415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.038965    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:27:53.063642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.063607    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 21:27:53.064142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.064108    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 21:27:53.099588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.099504    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 21:27:53.147552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.147513    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 21:27:53.177299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.177186    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:53.309443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.290707    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:27:53.309443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.290787    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 21:27:53.499355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.499322    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 21:27:53.519442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.519327    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 21:27:53.656470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.645466    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:27:53.672538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.672489    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:53.776475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.775495    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:27:53.818499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.807494    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:53.844418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.820483    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:27:53.918739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.918700    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:53.935419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.919999    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:53.946739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.946696    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:27:53.979426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:53.965045    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 21:27:54.090511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.061223    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 21:27:54.127510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.127466    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:27:54.199081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.186371    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 21:27:54.199382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.199351    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 21:27:54.199534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.186481    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:54.359475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.305434    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"]
Oct 30 21:27:54.396417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.396280    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:27:54.396417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.396345    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 21:27:54.533209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.533172    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 21:27:54.534297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.534262    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 21:27:54.658881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.658780    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:54.660014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.659986    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"]
Oct 30 21:27:54.680535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.680509    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"
Oct 30 21:27:54.692416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.686755    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:54.695693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.695665    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-27804d\""
Oct 30 21:27:54.738490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.738453    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"]
Oct 30 21:27:54.738835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.738814    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"
Oct 30 21:27:54.764791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.756466    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:54.764791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.764552    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-27804d-dockercfg-bg8n8\""
Oct 30 21:27:54.793413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.782783    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:54.793413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.789942    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"]
Oct 30 21:27:54.834436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.795558    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldbm\" (UniqueName: \"kubernetes.io/projected/08b09d3f-a61b-4812-b128-c4ea5cea422f-kube-api-access-jldbm\") pod \"payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg\" (UID: \"08b09d3f-a61b-4812-b128-c4ea5cea422f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"
Oct 30 21:27:54.834436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.795707    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791182e3-1f2f-400d-8577-fa86d6b74162-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9\" (UID: \"791182e3-1f2f-400d-8577-fa86d6b74162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"
Oct 30 21:27:54.834436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.795841    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b09d3f-a61b-4812-b128-c4ea5cea422f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg\" (UID: \"08b09d3f-a61b-4812-b128-c4ea5cea422f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"
Oct 30 21:27:54.834436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.796001    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdm2j\" (UniqueName: \"kubernetes.io/projected/791182e3-1f2f-400d-8577-fa86d6b74162-kube-api-access-vdm2j\") pod \"payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9\" (UID: \"791182e3-1f2f-400d-8577-fa86d6b74162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"
Oct 30 21:27:54.834436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.809459    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:54.850570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.850521    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:54.928262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.910587    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b09d3f-a61b-4812-b128-c4ea5cea422f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg\" (UID: \"08b09d3f-a61b-4812-b128-c4ea5cea422f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"
Oct 30 21:27:54.928262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.910770    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdm2j\" (UniqueName: \"kubernetes.io/projected/791182e3-1f2f-400d-8577-fa86d6b74162-kube-api-access-vdm2j\") pod \"payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9\" (UID: \"791182e3-1f2f-400d-8577-fa86d6b74162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"
Oct 30 21:27:54.928262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.910818    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldbm\" (UniqueName: \"kubernetes.io/projected/08b09d3f-a61b-4812-b128-c4ea5cea422f-kube-api-access-jldbm\") pod \"payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg\" (UID: \"08b09d3f-a61b-4812-b128-c4ea5cea422f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"
Oct 30 21:27:54.928262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.910946    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791182e3-1f2f-400d-8577-fa86d6b74162-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9\" (UID: \"791182e3-1f2f-400d-8577-fa86d6b74162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"
Oct 30 21:27:54.928262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.912044    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791182e3-1f2f-400d-8577-fa86d6b74162-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9\" (UID: \"791182e3-1f2f-400d-8577-fa86d6b74162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"
Oct 30 21:27:54.928262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.925703    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:27:54.929586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.929516    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b09d3f-a61b-4812-b128-c4ea5cea422f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg\" (UID: \"08b09d3f-a61b-4812-b128-c4ea5cea422f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"
Oct 30 21:27:54.942537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.942497    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:54.961575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.960801    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 21:27:54.961575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.960861    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 21:27:54.961575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:54.961266    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:27:55.012380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.012344    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdm2j\" (UniqueName: \"kubernetes.io/projected/791182e3-1f2f-400d-8577-fa86d6b74162-kube-api-access-vdm2j\") pod \"payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9\" (UID: \"791182e3-1f2f-400d-8577-fa86d6b74162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"
Oct 30 21:27:55.043194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.043159    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldbm\" (UniqueName: \"kubernetes.io/projected/08b09d3f-a61b-4812-b128-c4ea5cea422f-kube-api-access-jldbm\") pod \"payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg\" (UID: \"08b09d3f-a61b-4812-b128-c4ea5cea422f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"
Oct 30 21:27:55.094805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.094766    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 21:27:55.095090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.095057    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 21:27:55.213482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.201936    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:55.235424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.235056    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"
Oct 30 21:27:55.258179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.258027    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:55.340419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.329314    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:27:55.345711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.345503    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 21:27:55.361066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.341925    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"
Oct 30 21:27:55.614614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.583025    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded"
Oct 30 21:27:55.661959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.661924    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 21:27:55.662638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.662568    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 21:27:55.679455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.667474    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Oct 30 21:27:55.679455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.671837    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:55.688511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.688469    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:55.769950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.761486    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:27:55.769950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.762085    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:27:55.784027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.783980    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:27:55.813314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.784289    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:55.813314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.812381    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:55.996429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.935815    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:27:55.996429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.944481    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:55.996429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.961598    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 21:27:55.996429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.961654    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 21:27:55.996429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.961723    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:55.997373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:55.997153    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Oct 30 21:27:56.065056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.064968    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:27:56.161427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.158734    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Oct 30 21:27:56.161427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.158786    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Oct 30 21:27:56.340057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.272953    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:27:56.340057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.273018    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 21:27:56.340057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.273457    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:56.545417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.512052    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 21:27:56.545417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.512118    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 21:27:56.680414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.678511    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:56.704785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.689294    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:56.881000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.879539    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:56.881000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.879570    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:56.881000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.879943    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:27:56.881000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.880285    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:56.881836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.881662    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:27:56.938969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.938878    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:56.965923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:56.965881    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:57.021840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.021792    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:27:57.553442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.083625    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 21:27:57.553442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.550535    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 21:27:57.553442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.305435    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:27:57.559600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.148181    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:27:57.561823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.338273    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Oct 30 21:27:57.561823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.559839    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Oct 30 21:27:57.702786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.702749    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"]
Oct 30 21:27:57.703261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.702949    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:57.703261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.703060    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:27:57.703261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.703134    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:27:57.703261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.703185    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:27:57.703261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.703234    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:27:57.703581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.703284    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:27:57.703581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.703337    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:27:57.703581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.703409    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:27:57.703581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.703462    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:27:57.703581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.703511    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:27:57.703581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.703568    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:27:57.703877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.703620    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:27:57.709478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.709427    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:27:57.709617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.709576    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:27:57.715368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.715347    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:27:57.929602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:57.929380    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"]
Oct 30 21:27:58.026168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.024965    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:58.028042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.027971    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:27:58.267918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.267888    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:58.268189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.268170    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:58.268933 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.268914    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:27:58.644914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.644883    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg" event={"ID":"08b09d3f-a61b-4812-b128-c4ea5cea422f","Type":"ContainerStarted","Data":"73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b"}
Oct 30 21:27:58.645140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.644938    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg" event={"ID":"08b09d3f-a61b-4812-b128-c4ea5cea422f","Type":"ContainerStarted","Data":"d230423931e29f31b47b063b3a371b2816724d5e84cd99e36d9e19da81ee0ae4"}
Oct 30 21:27:58.645140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.645011    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"
Oct 30 21:27:58.647157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.647129    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9" event={"ID":"791182e3-1f2f-400d-8577-fa86d6b74162","Type":"ContainerStarted","Data":"a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11"}
Oct 30 21:27:58.647305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.647173    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9" event={"ID":"791182e3-1f2f-400d-8577-fa86d6b74162","Type":"ContainerStarted","Data":"2be111c99fbfe54876f5382a0f296b7eab46019710428b2b284bc364f765eb19"}
Oct 30 21:27:58.647305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.647234    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"
Oct 30 21:27:58.685466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.685370    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg" podStartSLOduration=6.685357387 podStartE2EDuration="6.685357387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:58.683202521 +0000 UTC m=+1591.729761689" watchObservedRunningTime="2026-10-30 21:27:58.685357387 +0000 UTC m=+1591.731916564"
Oct 30 21:27:58.716421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:27:58.716363    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9" podStartSLOduration=6.71635014 podStartE2EDuration="6.71635014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:58.712120859 +0000 UTC m=+1591.758680017" watchObservedRunningTime="2026-10-30 21:27:58.71635014 +0000 UTC m=+1591.762909287"
Oct 30 21:27:59.161267 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:27:59.161231    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:28:09.660423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:09.660376    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"
Oct 30 21:28:09.660936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:09.660883    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"
Oct 30 21:28:18.054449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.054421    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"]
Oct 30 21:28:18.059623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.059603    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.084425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.084361    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"]
Oct 30 21:28:18.149955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.149918    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.150148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.149964    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4849090-13cf-4a35-8160-86860f71d19c-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.150148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.150099    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.150285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.150227    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.150437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.150380    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.150437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.150428    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.150575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.150536    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.150635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.150604    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74g5j\" (UniqueName: \"kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-kube-api-access-74g5j\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.150678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.150651    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4849090-13cf-4a35-8160-86860f71d19c-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.252556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.252523    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.252556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.252562    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.252827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.252629    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.252827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.252693    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74g5j\" (UniqueName: \"kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-kube-api-access-74g5j\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.252827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.252752    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4849090-13cf-4a35-8160-86860f71d19c-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.252827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.252801    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.253050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.252843    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4849090-13cf-4a35-8160-86860f71d19c-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.253050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.252931    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.253050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.252949    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.253223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.253052    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.253223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.253058    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.253223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.253071    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.253370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.253286    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.253971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.253947    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4849090-13cf-4a35-8160-86860f71d19c-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.255352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.255332    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.255792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.255770    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4849090-13cf-4a35-8160-86860f71d19c-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.267281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.267252    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74g5j\" (UniqueName: \"kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-kube-api-access-74g5j\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.269123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.269098    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.382678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.382601    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:18.696989 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:28:18.696960    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4849090_13cf_4a35_8160_86860f71d19c.slice/crio-1b6b33451382a6e7f78496e9b4ab10430a97fc53f799f8a7d84592d5518e331b WatchSource:0}: Error finding container 1b6b33451382a6e7f78496e9b4ab10430a97fc53f799f8a7d84592d5518e331b: Status 404 returned error can't find the container with id 1b6b33451382a6e7f78496e9b4ab10430a97fc53f799f8a7d84592d5518e331b
Oct 30 21:28:18.697218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.697196    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"]
Oct 30 21:28:18.700222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.700186    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:28:18.700310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.700269    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:28:18.700366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:18.700309    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:28:19.052474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:19.052437    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99" event={"ID":"a4849090-13cf-4a35-8160-86860f71d19c","Type":"ContainerStarted","Data":"11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573"}
Oct 30 21:28:19.052662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:19.052502    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99" event={"ID":"a4849090-13cf-4a35-8160-86860f71d19c","Type":"ContainerStarted","Data":"1b6b33451382a6e7f78496e9b4ab10430a97fc53f799f8a7d84592d5518e331b"}
Oct 30 21:28:19.093064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:19.093016    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99" podStartSLOduration=2.09300216 podStartE2EDuration="2.09300216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:19.090640782 +0000 UTC m=+1612.137199926" watchObservedRunningTime="2026-10-30 21:28:19.09300216 +0000 UTC m=+1612.139561304"
Oct 30 21:28:19.382867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:19.382784    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:19.384342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:19.384319    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:28:19.384480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:19.384375    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99" podUID="a4849090-13cf-4a35-8160-86860f71d19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Oct 30 21:28:20.383411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:20.383305    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:28:20.383411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:20.383363    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99" podUID="a4849090-13cf-4a35-8160-86860f71d19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Oct 30 21:28:21.383352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:21.383315    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:28:21.383555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:21.383377    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99" podUID="a4849090-13cf-4a35-8160-86860f71d19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Oct 30 21:28:22.383223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:22.383193    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:28:22.383383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:22.383249    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99" podUID="a4849090-13cf-4a35-8160-86860f71d19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Oct 30 21:28:23.382953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:23.382916    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:28:23.383351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:23.382969    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99" podUID="a4849090-13cf-4a35-8160-86860f71d19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Oct 30 21:28:24.383276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:24.383239    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Oct 30 21:28:24.383728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:24.383310    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99" podUID="a4849090-13cf-4a35-8160-86860f71d19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Oct 30 21:28:25.389416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:25.389368    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:25.389796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:25.389677    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:25.390406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:25.390369    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:28:29.160750 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:28:29.160718    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:28:30.540551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:30.540515    4918 scope.go:122] "RemoveContainer" containerID="98bc43ff2044a5ec0f9f702e4175bb1bc1f0d41a5c870b23392107a5647ada54"
Oct 30 21:28:30.552340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:30.552319    4918 scope.go:122] "RemoveContainer" containerID="be378ec323ee9c167a0cf61c3f78268ae6397a501d924a1bec2febafb8cb776a"
Oct 30 21:28:30.562637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:30.562611    4918 scope.go:122] "RemoveContainer" containerID="baa974c152096eea6cbdc29c958a198f8cdfff7b6ecbc0d54b7c62522da4faa1"
Oct 30 21:28:33.110430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.109299    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"]
Oct 30 21:28:33.116060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.116027    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.135628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.135594    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"]
Oct 30 21:28:33.256732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.256688    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.256940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.256770    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.256940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.256816    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.256940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.256870    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.257123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.256990    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.257189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.257173    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.257243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.257228    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjjmr\" (UniqueName: \"kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-kube-api-access-hjjmr\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.257298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.257264    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.257348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.257318    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.372467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.363474    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.372467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.363553    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjjmr\" (UniqueName: \"kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-kube-api-access-hjjmr\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.372467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.363585    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.372467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.363630    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.372467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.363713    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.372467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.363758    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.372467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.363790    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.372467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.363835    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.372467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.363951    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.372467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.364534    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.372467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.364786    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-certs\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.393424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.379314    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-data\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.393424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.391910    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istiod-ca-cert\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.393424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.392146    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-credential-socket\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.394242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.394213    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-envoy\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.424413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.417173    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjjmr\" (UniqueName: \"kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-kube-api-access-hjjmr\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.425081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.425051    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-token\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.425275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.425143    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-podinfo\") pod \"e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") " pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:33.441826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:33.436688    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:34.561828 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:28:34.561790    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c783f14_4102_4a9a_a62d_4c971d7dd31b.slice/crio-ebf576c86629fac8cec113307ef5bad79b93681e665ca7725320139bb8817289 WatchSource:0}: Error finding container ebf576c86629fac8cec113307ef5bad79b93681e665ca7725320139bb8817289: Status 404 returned error can't find the container with id ebf576c86629fac8cec113307ef5bad79b93681e665ca7725320139bb8817289
Oct 30 21:28:34.562920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:34.562898    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"]
Oct 30 21:28:34.564862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:34.564829    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:28:34.564951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:34.564896    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:28:34.564951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:34.564925    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:28:35.423679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:35.423646    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw" event={"ID":"1c783f14-4102-4a9a-a62d-4c971d7dd31b","Type":"ContainerStarted","Data":"e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5"}
Oct 30 21:28:35.423924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:35.423712    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw" event={"ID":"1c783f14-4102-4a9a-a62d-4c971d7dd31b","Type":"ContainerStarted","Data":"ebf576c86629fac8cec113307ef5bad79b93681e665ca7725320139bb8817289"}
Oct 30 21:28:35.438062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:35.438038    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:35.439957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:35.439929    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 30 21:28:35.440058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:35.439999    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw" podUID="1c783f14-4102-4a9a-a62d-4c971d7dd31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 30 21:28:35.516413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:35.515805    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw" podStartSLOduration=2.515782047 podStartE2EDuration="2.515782047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:35.476777539 +0000 UTC m=+1628.523336756" watchObservedRunningTime="2026-10-30 21:28:35.515782047 +0000 UTC m=+1628.562341193"
Oct 30 21:28:36.437312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:36.437279    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 30 21:28:36.437843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:36.437333    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw" podUID="1c783f14-4102-4a9a-a62d-4c971d7dd31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 30 21:28:37.437631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:37.437585    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 30 21:28:37.438073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:37.437662    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw" podUID="1c783f14-4102-4a9a-a62d-4c971d7dd31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 30 21:28:38.437638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:38.437602    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 30 21:28:38.438082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:38.437667    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw" podUID="1c783f14-4102-4a9a-a62d-4c971d7dd31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 30 21:28:39.438179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:39.438144    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 30 21:28:39.438687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:39.438202    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw" podUID="1c783f14-4102-4a9a-a62d-4c971d7dd31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 30 21:28:40.438207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:40.438169    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Oct 30 21:28:40.438816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:40.438234    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw" podUID="1c783f14-4102-4a9a-a62d-4c971d7dd31b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Oct 30 21:28:41.442339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:41.442309    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:41.442886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:41.442816    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:41.443698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:41.443678    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:28:47.354425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.354376    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"]
Oct 30 21:28:47.359287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.359265    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.379790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.379769    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-15e79c-openshift-default-dockercfg-vhqh5\""
Oct 30 21:28:47.464715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.464684    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"]
Oct 30 21:28:47.535918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.535885    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-socket\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.536094 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.535945    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-envoy\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.536094 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.536059    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9954z\" (UniqueName: \"kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-kube-api-access-9954z\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.536213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.536119    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-credential-socket\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.536272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.536211    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-podinfo\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.536335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.536316    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debec1c0-11e0-464c-a798-4cb6f334eb32-istiod-ca-cert\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.536500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.536482    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-data\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.536588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.536522    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-token\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.536655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.536640    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-certs\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.638170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638089    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-certs\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.638170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638163    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-socket\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.638451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638217    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-envoy\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.638451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638310    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9954z\" (UniqueName: \"kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-kube-api-access-9954z\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.638451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638437    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-credential-socket\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.638644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638477    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-podinfo\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.638644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638534    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debec1c0-11e0-464c-a798-4cb6f334eb32-istiod-ca-cert\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.638644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638573    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-certs\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.638644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638615    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-socket\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.638871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638658    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-data\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.638871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638693    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-token\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.638871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638830    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-credential-socket\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.639045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.638961    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-data\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.639227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.639206    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debec1c0-11e0-464c-a798-4cb6f334eb32-istiod-ca-cert\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.640745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.640725    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-envoy\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.641062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.641044    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-podinfo\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.648648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.648628    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9954z\" (UniqueName: \"kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-kube-api-access-9954z\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.651678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.651655    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-token\") pod \"e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") " pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.675096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.675076    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:47.997035 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:28:47.997003    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebec1c0_11e0_464c_a798_4cb6f334eb32.slice/crio-6484e4f0afda6dbe2c0d3b70fe5fa42551a7bc4bf6f32a92df8badd3d86af239 WatchSource:0}: Error finding container 6484e4f0afda6dbe2c0d3b70fe5fa42551a7bc4bf6f32a92df8badd3d86af239: Status 404 returned error can't find the container with id 6484e4f0afda6dbe2c0d3b70fe5fa42551a7bc4bf6f32a92df8badd3d86af239
Oct 30 21:28:47.997417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.997375    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"]
Oct 30 21:28:47.999738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.999707    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:28:47.999837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.999775    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:28:47.999837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:47.999805    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:28:48.538621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:48.538589    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6" event={"ID":"debec1c0-11e0-464c-a798-4cb6f334eb32","Type":"ContainerStarted","Data":"d667bfc9190a02448fa011b20c8cdd7e61c86a3b3051e6bdc916969767794d1a"}
Oct 30 21:28:48.539009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:48.538652    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6" event={"ID":"debec1c0-11e0-464c-a798-4cb6f334eb32","Type":"ContainerStarted","Data":"6484e4f0afda6dbe2c0d3b70fe5fa42551a7bc4bf6f32a92df8badd3d86af239"}
Oct 30 21:28:48.593888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:48.593822    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6" podStartSLOduration=1.593790046 podStartE2EDuration="1.593790046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:48.589085921 +0000 UTC m=+1641.635645087" watchObservedRunningTime="2026-10-30 21:28:48.593790046 +0000 UTC m=+1641.640349192"
Oct 30 21:28:48.683508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:48.680638    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:49.063400 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:49.063353    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"]
Oct 30 21:28:49.084363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:49.084335    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b8ef01-pzl5h"]
Oct 30 21:28:49.677191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:49.677159    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:28:49.677595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:49.677215    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6" podUID="debec1c0-11e0-464c-a798-4cb6f334eb32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 21:28:50.676056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:50.676025    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:28:50.676259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:50.676078    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6" podUID="debec1c0-11e0-464c-a798-4cb6f334eb32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 21:28:51.115817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:51.115779    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8352713-f7bb-4055-a810-b969caf48a11" path="/var/lib/kubelet/pods/e8352713-f7bb-4055-a810-b969caf48a11/volumes"
Oct 30 21:28:51.675970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:51.675935    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:28:51.676260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:51.676229    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6" podUID="debec1c0-11e0-464c-a798-4cb6f334eb32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 21:28:52.677842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:52.677804    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:28:52.678496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:52.678464    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6" podUID="debec1c0-11e0-464c-a798-4cb6f334eb32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 21:28:53.687375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:53.687336    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:28:53.688030 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:53.687989    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6" podUID="debec1c0-11e0-464c-a798-4cb6f334eb32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 21:28:54.686914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:54.686878    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 30 21:28:54.687288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:54.687250    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6" podUID="debec1c0-11e0-464c-a798-4cb6f334eb32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 30 21:28:55.797521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:55.797489    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:55.798084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:55.798064    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:55.799081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:28:55.799058    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:28:59.160930 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:28:59.160896    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:29:05.050757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:05.050713    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"]
Oct 30 21:29:05.062844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:05.062817    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59a10a62-ntpm8"]
Oct 30 21:29:05.079810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:05.079784    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278ba805-4821-4ba9-92be-4c02614d0380" path="/var/lib/kubelet/pods/278ba805-4821-4ba9-92be-4c02614d0380/volumes"
Oct 30 21:29:09.301501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.301293    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"]
Oct 30 21:29:09.311383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.311352    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.345003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.344975    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-46c98c-openshift-default-dockercfg-tmsn4\""
Oct 30 21:29:09.354499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.354471    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"]
Oct 30 21:29:09.538447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.537494    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-envoy\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.538447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.537662    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl4r8\" (UniqueName: \"kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-kube-api-access-hl4r8\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.706672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.689320    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-data\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.706672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.689453    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-socket\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.706672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.689526    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istiod-ca-cert\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.706672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.689572    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-certs\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.706672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.689604    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-credential-socket\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.706672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.689691    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-podinfo\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.706672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.689786    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-token\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.791840    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-podinfo\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.791938    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-token\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.792052    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-envoy\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.792187    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl4r8\" (UniqueName: \"kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-kube-api-access-hl4r8\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.792263    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-data\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.792327    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-socket\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.792382    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istiod-ca-cert\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.792439    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-certs\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.792470    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-credential-socket\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.793143    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-credential-socket\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.793986    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-data\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.795044    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istiod-ca-cert\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.795371    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-socket\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.795632    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-certs\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.815487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.812937    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-envoy\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.815487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.814250    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-podinfo\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.824093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.824029    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-token\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.829236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.829207    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl4r8\" (UniqueName: \"kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-kube-api-access-hl4r8\") pod \"e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") " pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:09.927188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:09.927155    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:10.299441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:10.299407    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"]
Oct 30 21:29:10.303116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:10.303082    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:29:10.303560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:10.303165    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:29:10.303560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:10.303215    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:29:11.161850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:11.161815    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" event={"ID":"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2","Type":"ContainerStarted","Data":"4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80"}
Oct 30 21:29:11.162142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:11.161873    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" event={"ID":"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2","Type":"ContainerStarted","Data":"0261a26ca24bd34e0d6da146b81806c7ce44283c9b9c8bfb47a6db750bc94561"}
Oct 30 21:29:11.249749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:11.249700    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" podStartSLOduration=2.249685554 podStartE2EDuration="2.249685554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:11.24359549 +0000 UTC m=+1664.290154634" watchObservedRunningTime="2026-10-30 21:29:11.249685554 +0000 UTC m=+1664.296244699"
Oct 30 21:29:11.927857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:11.927818    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:11.933142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:11.933118    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:12.170845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:12.170815    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:12.171906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:12.171887    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:29:15.492251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:15.492214    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"]
Oct 30 21:29:15.497682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:15.497659    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"
Oct 30 21:29:15.509116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:15.509093    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-46c98c\""
Oct 30 21:29:15.553054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:15.553022    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2jw2\" (UniqueName: \"kubernetes.io/projected/312757a6-8779-4a73-8e59-389d4a7dbb91-kube-api-access-q2jw2\") pod \"payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj\" (UID: \"312757a6-8779-4a73-8e59-389d4a7dbb91\") " pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"
Oct 30 21:29:15.553244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:15.553068    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312757a6-8779-4a73-8e59-389d4a7dbb91-plugins-config-volume\") pod \"payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj\" (UID: \"312757a6-8779-4a73-8e59-389d4a7dbb91\") " pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"
Oct 30 21:29:15.591489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:15.591459    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"]
Oct 30 21:29:15.660240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:15.660190    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2jw2\" (UniqueName: \"kubernetes.io/projected/312757a6-8779-4a73-8e59-389d4a7dbb91-kube-api-access-q2jw2\") pod \"payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj\" (UID: \"312757a6-8779-4a73-8e59-389d4a7dbb91\") " pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"
Oct 30 21:29:15.660571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:15.660544    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312757a6-8779-4a73-8e59-389d4a7dbb91-plugins-config-volume\") pod \"payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj\" (UID: \"312757a6-8779-4a73-8e59-389d4a7dbb91\") " pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"
Oct 30 21:29:15.661824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:15.661791    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312757a6-8779-4a73-8e59-389d4a7dbb91-plugins-config-volume\") pod \"payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj\" (UID: \"312757a6-8779-4a73-8e59-389d4a7dbb91\") " pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"
Oct 30 21:29:15.771465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:15.734309    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2jw2\" (UniqueName: \"kubernetes.io/projected/312757a6-8779-4a73-8e59-389d4a7dbb91-kube-api-access-q2jw2\") pod \"payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj\" (UID: \"312757a6-8779-4a73-8e59-389d4a7dbb91\") " pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"
Oct 30 21:29:15.825414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:15.821699    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"
Oct 30 21:29:16.313286 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:29:16.313257    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312757a6_8779_4a73_8e59_389d4a7dbb91.slice/crio-d33ea1802c9c7ca2be6e218da63f658ea80f68a962bbfc1fe7e73e536f1fd2fc WatchSource:0}: Error finding container d33ea1802c9c7ca2be6e218da63f658ea80f68a962bbfc1fe7e73e536f1fd2fc: Status 404 returned error can't find the container with id d33ea1802c9c7ca2be6e218da63f658ea80f68a962bbfc1fe7e73e536f1fd2fc
Oct 30 21:29:16.315305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:16.315283    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"]
Oct 30 21:29:17.271913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:17.271869    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj" event={"ID":"312757a6-8779-4a73-8e59-389d4a7dbb91","Type":"ContainerStarted","Data":"5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c"}
Oct 30 21:29:17.272468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:17.271959    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"
Oct 30 21:29:17.272468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:17.271997    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj" event={"ID":"312757a6-8779-4a73-8e59-389d4a7dbb91","Type":"ContainerStarted","Data":"d33ea1802c9c7ca2be6e218da63f658ea80f68a962bbfc1fe7e73e536f1fd2fc"}
Oct 30 21:29:17.469889 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:17.469822    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj" podStartSLOduration=2.469800608 podStartE2EDuration="2.469800608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:17.465464914 +0000 UTC m=+1670.512024060" watchObservedRunningTime="2026-10-30 21:29:17.469800608 +0000 UTC m=+1670.516359783"
Oct 30 21:29:19.054331 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:19.054298    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"]
Oct 30 21:29:19.059327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:19.059299    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6decbf37-t8vfq"]
Oct 30 21:29:19.079220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:19.079188    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe94b2d-0a3e-4063-b3d5-e8a88ef28119" path="/var/lib/kubelet/pods/6fe94b2d-0a3e-4063-b3d5-e8a88ef28119/volumes"
Oct 30 21:29:28.283223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:28.283195    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"
Oct 30 21:29:29.161610 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:29:29.161575    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:29:30.060726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:30.060688    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"]
Oct 30 21:29:30.069055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:30.069031    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141fb921-zzr7j"]
Oct 30 21:29:30.607575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:30.607549    4918 scope.go:122] "RemoveContainer" containerID="143cef77a95d989c40081c47c5cbda1c5d9dd7e1f9521965fa4fb5e97b7cfddb"
Oct 30 21:29:30.619185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:30.619168    4918 scope.go:122] "RemoveContainer" containerID="a57f143bb285b8ac0923bb1c3d40b171f73bee39df97c3d8a6c72931e30c09b8"
Oct 30 21:29:30.629494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:30.629470    4918 scope.go:122] "RemoveContainer" containerID="85373d4736dddc57b88924bb306e6646f0f322a47dd0b4c54f8063a36f00a2f7"
Oct 30 21:29:30.639514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:30.639498    4918 scope.go:122] "RemoveContainer" containerID="034579598db6d3c8827222da9c7475df0547efe9b9bc668e46145536ce1ee248"
Oct 30 21:29:30.649162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:30.649147    4918 scope.go:122] "RemoveContainer" containerID="708d4f973f9eff87ba2ac195bf894e09c889abf49e9d2e0cfd3be20633594e87"
Oct 30 21:29:31.079942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:31.079912    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df5523d-2d07-43ba-84d8-4b6bcd947a80" path="/var/lib/kubelet/pods/5df5523d-2d07-43ba-84d8-4b6bcd947a80/volumes"
Oct 30 21:29:40.060227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:40.060153    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"]
Oct 30 21:29:40.073521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:40.073482    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-004c94a0-bp572"]
Oct 30 21:29:41.079332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:41.079298    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e408ab-ce1b-40b6-8282-0f12715ff41c" path="/var/lib/kubelet/pods/22e408ab-ce1b-40b6-8282-0f12715ff41c/volumes"
Oct 30 21:29:44.029884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:44.029845    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"]
Oct 30 21:29:44.037798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:44.037766    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dba79e2-6dvwg"]
Oct 30 21:29:45.081721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:45.081688    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d93c9ac-2a73-47d3-9304-da0a51852b65" path="/var/lib/kubelet/pods/5d93c9ac-2a73-47d3-9304-da0a51852b65/volumes"
Oct 30 21:29:46.150216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:46.150184    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"]
Oct 30 21:29:46.150768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:46.150404    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" podUID="a0439b0d-ff5f-49ab-87c3-afada363adcb" containerName="istio-proxy" containerID="cri-o://a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd" gracePeriod=30
Oct 30 21:29:51.439381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.439349    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:29:51.521772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.521722    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-certs\") pod \"a0439b0d-ff5f-49ab-87c3-afada363adcb\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") "
Oct 30 21:29:51.521950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.521795    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0439b0d-ff5f-49ab-87c3-afada363adcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0439b0d-ff5f-49ab-87c3-afada363adcb-istiod-ca-cert\") pod \"a0439b0d-ff5f-49ab-87c3-afada363adcb\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") "
Oct 30 21:29:51.521950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.521862    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-envoy\") pod \"a0439b0d-ff5f-49ab-87c3-afada363adcb\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") "
Oct 30 21:29:51.522052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.521961    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-socket\") pod \"a0439b0d-ff5f-49ab-87c3-afada363adcb\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") "
Oct 30 21:29:51.522052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.521985    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-certs" pod "a0439b0d-ff5f-49ab-87c3-afada363adcb" (UID: "a0439b0d-ff5f-49ab-87c3-afada363adcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:51.522175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.522133    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-podinfo\") pod \"a0439b0d-ff5f-49ab-87c3-afada363adcb\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") "
Oct 30 21:29:51.522175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.522166    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0439b0d-ff5f-49ab-87c3-afada363adcb-istiod-ca-cert" pod "a0439b0d-ff5f-49ab-87c3-afada363adcb" (UID: "a0439b0d-ff5f-49ab-87c3-afada363adcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:29:51.522291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.522205    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-token\" (UniqueName: \"kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-token\") pod \"a0439b0d-ff5f-49ab-87c3-afada363adcb\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") "
Oct 30 21:29:51.522291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.522276    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-credential-socket\") pod \"a0439b0d-ff5f-49ab-87c3-afada363adcb\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") "
Oct 30 21:29:51.522291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.522274    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-socket" pod "a0439b0d-ff5f-49ab-87c3-afada363adcb" (UID: "a0439b0d-ff5f-49ab-87c3-afada363adcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:51.522582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.522316    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-data\") pod \"a0439b0d-ff5f-49ab-87c3-afada363adcb\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") "
Oct 30 21:29:51.522582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.522415    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-kube-api-access-fgtqr\" (UniqueName: \"kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-kube-api-access-fgtqr\") pod \"a0439b0d-ff5f-49ab-87c3-afada363adcb\" (UID: \"a0439b0d-ff5f-49ab-87c3-afada363adcb\") "
Oct 30 21:29:51.522840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.522816    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-data" pod "a0439b0d-ff5f-49ab-87c3-afada363adcb" (UID: "a0439b0d-ff5f-49ab-87c3-afada363adcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:51.522983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.522835    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-credential-socket" pod "a0439b0d-ff5f-49ab-87c3-afada363adcb" (UID: "a0439b0d-ff5f-49ab-87c3-afada363adcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:51.523169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.523152    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:51.523258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.523171    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:51.523258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.523181    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0439b0d-ff5f-49ab-87c3-afada363adcb-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:51.523258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.523189    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:51.524639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.524615    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-envoy" pod "a0439b0d-ff5f-49ab-87c3-afada363adcb" (UID: "a0439b0d-ff5f-49ab-87c3-afada363adcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:29:51.524745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.524658    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-podinfo" pod "a0439b0d-ff5f-49ab-87c3-afada363adcb" (UID: "a0439b0d-ff5f-49ab-87c3-afada363adcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:29:51.525232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.525215    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-kube-api-access-fgtqr" pod "a0439b0d-ff5f-49ab-87c3-afada363adcb" (UID: "a0439b0d-ff5f-49ab-87c3-afada363adcb"). InnerVolumeSpecName "kube-api-access-fgtqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:51.525316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.525241    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-token" pod "a0439b0d-ff5f-49ab-87c3-afada363adcb" (UID: "a0439b0d-ff5f-49ab-87c3-afada363adcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:29:51.624284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.624253    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgtqr\" (UniqueName: \"kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-kube-api-access-fgtqr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:51.624284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.624281    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:51.624284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.624291    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:51.624550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.624298    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0439b0d-ff5f-49ab-87c3-afada363adcb-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:51.624550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.624307    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0439b0d-ff5f-49ab-87c3-afada363adcb-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:29:51.707608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.707518    4918 generic.go:356] "Generic (PLEG): container finished" podID="a0439b0d-ff5f-49ab-87c3-afada363adcb" containerID="a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd" exitCode=0
Oct 30 21:29:51.707608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.707590    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"
Oct 30 21:29:51.707805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.707606    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" event={"ID":"a0439b0d-ff5f-49ab-87c3-afada363adcb","Type":"ContainerDied","Data":"a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd"}
Oct 30 21:29:51.707805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.707698    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz" event={"ID":"a0439b0d-ff5f-49ab-87c3-afada363adcb","Type":"ContainerDied","Data":"070351413d354aa5c89096fd0b037853bd98e9d2835ce075dc3b0979305f2be1"}
Oct 30 21:29:51.707805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.707727    4918 scope.go:122] "RemoveContainer" containerID="a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd"
Oct 30 21:29:51.721347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.721322    4918 scope.go:122] "RemoveContainer" containerID="a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd"
Oct 30 21:29:51.721663 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:29:51.721643    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd\": container with ID starting with a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd not found: ID does not exist" containerID="a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd"
Oct 30 21:29:51.721719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.721671    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd"} err="failed to get container status \"a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd\": rpc error: code = NotFound desc = could not find container \"a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd\": container with ID starting with a24adc6b291e838bf2e21f6bbca346b7c07bc6594855626b63dea122847ee8dd not found: ID does not exist"
Oct 30 21:29:51.744542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.744512    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"]
Oct 30 21:29:51.756840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:51.756797    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-smshz"]
Oct 30 21:29:53.080794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:53.080766    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0439b0d-ff5f-49ab-87c3-afada363adcb" path="/var/lib/kubelet/pods/a0439b0d-ff5f-49ab-87c3-afada363adcb/volumes"
Oct 30 21:29:56.052517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:56.052475    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"]
Oct 30 21:29:56.063416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:56.061658    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d2696f-8mqmr"]
Oct 30 21:29:57.080580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:29:57.080549    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5139dc2e-8d18-4d96-b8d4-c91d6d9e6096" path="/var/lib/kubelet/pods/5139dc2e-8d18-4d96-b8d4-c91d6d9e6096/volumes"
Oct 30 21:29:59.294877 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:29:59.294836    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:30:00.192418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.191942    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"]
Oct 30 21:30:00.202450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.193256    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a0439b0d-ff5f-49ab-87c3-afada363adcb" containerName="istio-proxy"
Oct 30 21:30:00.202450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.193282    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0439b0d-ff5f-49ab-87c3-afada363adcb" containerName="istio-proxy"
Oct 30 21:30:00.202450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.193544    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0439b0d-ff5f-49ab-87c3-afada363adcb" containerName="istio-proxy"
Oct 30 21:30:00.221497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.209050    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"
Oct 30 21:30:00.230486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.228042    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"]
Oct 30 21:30:00.252033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.246057    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:30:00.252033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.246170    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:30:00.252033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.246258    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:30:00.261410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.258322    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"
Oct 30 21:30:00.285782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.285753    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"]
Oct 30 21:30:00.358211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.346708    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"
Oct 30 21:30:00.407418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.393519    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/778db27c-4f22-4317-9753-b84985424b0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz\" (UID: \"778db27c-4f22-4317-9753-b84985424b0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"
Oct 30 21:30:00.407418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.393630    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q8x8\" (UniqueName: \"kubernetes.io/projected/778db27c-4f22-4317-9753-b84985424b0b-kube-api-access-7q8x8\") pod \"maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz\" (UID: \"778db27c-4f22-4317-9753-b84985424b0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"
Oct 30 21:30:00.407418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.393669    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc27f4e0-65dd-4971-abee-816ac50c9869-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v\" (UID: \"fc27f4e0-65dd-4971-abee-816ac50c9869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"
Oct 30 21:30:00.407418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.393864    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n7rn\" (UniqueName: \"kubernetes.io/projected/fc27f4e0-65dd-4971-abee-816ac50c9869-kube-api-access-7n7rn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v\" (UID: \"fc27f4e0-65dd-4971-abee-816ac50c9869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"
Oct 30 21:30:00.504208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.495644    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfpfh\" (UniqueName: \"kubernetes.io/projected/9fbca41f-798b-4b86-b103-47868642c624-kube-api-access-qfpfh\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct\" (UID: \"9fbca41f-798b-4b86-b103-47868642c624\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"
Oct 30 21:30:00.504208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.495753    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/778db27c-4f22-4317-9753-b84985424b0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz\" (UID: \"778db27c-4f22-4317-9753-b84985424b0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"
Oct 30 21:30:00.504208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.495802    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbca41f-798b-4b86-b103-47868642c624-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct\" (UID: \"9fbca41f-798b-4b86-b103-47868642c624\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"
Oct 30 21:30:00.504208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.495867    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8x8\" (UniqueName: \"kubernetes.io/projected/778db27c-4f22-4317-9753-b84985424b0b-kube-api-access-7q8x8\") pod \"maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz\" (UID: \"778db27c-4f22-4317-9753-b84985424b0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"
Oct 30 21:30:00.504208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.495896    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc27f4e0-65dd-4971-abee-816ac50c9869-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v\" (UID: \"fc27f4e0-65dd-4971-abee-816ac50c9869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"
Oct 30 21:30:00.504208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.496085    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n7rn\" (UniqueName: \"kubernetes.io/projected/fc27f4e0-65dd-4971-abee-816ac50c9869-kube-api-access-7n7rn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v\" (UID: \"fc27f4e0-65dd-4971-abee-816ac50c9869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"
Oct 30 21:30:00.504208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.497368    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/778db27c-4f22-4317-9753-b84985424b0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz\" (UID: \"778db27c-4f22-4317-9753-b84985424b0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"
Oct 30 21:30:00.523270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.506485    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc27f4e0-65dd-4971-abee-816ac50c9869-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v\" (UID: \"fc27f4e0-65dd-4971-abee-816ac50c9869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"
Oct 30 21:30:00.632767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.632728    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfpfh\" (UniqueName: \"kubernetes.io/projected/9fbca41f-798b-4b86-b103-47868642c624-kube-api-access-qfpfh\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct\" (UID: \"9fbca41f-798b-4b86-b103-47868642c624\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"
Oct 30 21:30:00.633150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.633124    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbca41f-798b-4b86-b103-47868642c624-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct\" (UID: \"9fbca41f-798b-4b86-b103-47868642c624\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"
Oct 30 21:30:00.634151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.634129    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbca41f-798b-4b86-b103-47868642c624-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct\" (UID: \"9fbca41f-798b-4b86-b103-47868642c624\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"
Oct 30 21:30:00.680728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.680674    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfpfh\" (UniqueName: \"kubernetes.io/projected/9fbca41f-798b-4b86-b103-47868642c624-kube-api-access-qfpfh\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct\" (UID: \"9fbca41f-798b-4b86-b103-47868642c624\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"
Oct 30 21:30:00.721905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.716925    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"]
Oct 30 21:30:00.744762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.744687    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"
Oct 30 21:30:00.780318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.780287    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"]
Oct 30 21:30:00.782208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.782175    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q8x8\" (UniqueName: \"kubernetes.io/projected/778db27c-4f22-4317-9753-b84985424b0b-kube-api-access-7q8x8\") pod \"maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz\" (UID: \"778db27c-4f22-4317-9753-b84985424b0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"
Oct 30 21:30:00.786573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.786546    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n7rn\" (UniqueName: \"kubernetes.io/projected/fc27f4e0-65dd-4971-abee-816ac50c9869-kube-api-access-7n7rn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v\" (UID: \"fc27f4e0-65dd-4971-abee-816ac50c9869\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"
Oct 30 21:30:00.791283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.791262    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"]
Oct 30 21:30:00.791623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.791606    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"
Oct 30 21:30:00.798511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.798484    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"]
Oct 30 21:30:00.801700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.801478    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"]
Oct 30 21:30:00.815714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.815678    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"]
Oct 30 21:30:00.827661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.827627    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"]
Oct 30 21:30:00.828171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.828147    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"
Oct 30 21:30:00.834434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.834410    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"
Oct 30 21:30:00.853543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.853510    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"]
Oct 30 21:30:00.864918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.864858    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8cf\" (UniqueName: \"kubernetes.io/projected/43a51867-ecfe-43e9-b630-b621e9b4e428-kube-api-access-ks8cf\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx\" (UID: \"43a51867-ecfe-43e9-b630-b621e9b4e428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"
Oct 30 21:30:00.864918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.864920    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wvv\" (UniqueName: \"kubernetes.io/projected/1a6c38be-7c77-435c-a127-3d3fd054574e-kube-api-access-z6wvv\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65\" (UID: \"1a6c38be-7c77-435c-a127-3d3fd054574e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"
Oct 30 21:30:00.865173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.864997    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a51867-ecfe-43e9-b630-b621e9b4e428-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx\" (UID: \"43a51867-ecfe-43e9-b630-b621e9b4e428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"
Oct 30 21:30:00.865173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.865082    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt\" (UID: \"e1225dbb-da72-4c3f-b13d-b63d71e3ec51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"
Oct 30 21:30:00.865173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.865130    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67kc\" (UniqueName: \"kubernetes.io/projected/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-kube-api-access-l67kc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt\" (UID: \"e1225dbb-da72-4c3f-b13d-b63d71e3ec51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"
Oct 30 21:30:00.865354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.865182    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jhfp\" (UniqueName: \"kubernetes.io/projected/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-kube-api-access-7jhfp\") pod \"maas-api-key-cleanup-29889930-kwv29\" (UID: \"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"
Oct 30 21:30:00.865354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.865304    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-kwv29\" (UID: \"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"
Oct 30 21:30:00.865566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.865366    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6c38be-7c77-435c-a127-3d3fd054574e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65\" (UID: \"1a6c38be-7c77-435c-a127-3d3fd054574e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"
Oct 30 21:30:00.875455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.871377    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"]
Oct 30 21:30:00.927422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.917213    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"
Oct 30 21:30:00.933340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.933306    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"
Oct 30 21:30:00.950901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.950566    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"
Oct 30 21:30:00.955503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.953167    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"]
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.966943    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.969256    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8cf\" (UniqueName: \"kubernetes.io/projected/43a51867-ecfe-43e9-b630-b621e9b4e428-kube-api-access-ks8cf\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx\" (UID: \"43a51867-ecfe-43e9-b630-b621e9b4e428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.969311    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wvv\" (UniqueName: \"kubernetes.io/projected/1a6c38be-7c77-435c-a127-3d3fd054574e-kube-api-access-z6wvv\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65\" (UID: \"1a6c38be-7c77-435c-a127-3d3fd054574e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.969383    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a51867-ecfe-43e9-b630-b621e9b4e428-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx\" (UID: \"43a51867-ecfe-43e9-b630-b621e9b4e428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.969482    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt\" (UID: \"e1225dbb-da72-4c3f-b13d-b63d71e3ec51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.969523    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67kc\" (UniqueName: \"kubernetes.io/projected/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-kube-api-access-l67kc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt\" (UID: \"e1225dbb-da72-4c3f-b13d-b63d71e3ec51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.969578    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jhfp\" (UniqueName: \"kubernetes.io/projected/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-kube-api-access-7jhfp\") pod \"maas-api-key-cleanup-29889930-kwv29\" (UID: \"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.969684    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-kwv29\" (UID: \"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.969731    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6c38be-7c77-435c-a127-3d3fd054574e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65\" (UID: \"1a6c38be-7c77-435c-a127-3d3fd054574e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.970735    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6c38be-7c77-435c-a127-3d3fd054574e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65\" (UID: \"1a6c38be-7c77-435c-a127-3d3fd054574e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.972005    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a51867-ecfe-43e9-b630-b621e9b4e428-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx\" (UID: \"43a51867-ecfe-43e9-b630-b621e9b4e428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.976006    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt\" (UID: \"e1225dbb-da72-4c3f-b13d-b63d71e3ec51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.977834    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"]
Oct 30 21:30:00.980262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:00.980081    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-kwv29\" (UID: \"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"
Oct 30 21:30:01.009253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.009169    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wvv\" (UniqueName: \"kubernetes.io/projected/1a6c38be-7c77-435c-a127-3d3fd054574e-kube-api-access-z6wvv\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65\" (UID: \"1a6c38be-7c77-435c-a127-3d3fd054574e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"
Oct 30 21:30:01.030509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.030473    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"]
Oct 30 21:30:01.041136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.040936    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"
Oct 30 21:30:01.053013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.052989    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"
Oct 30 21:30:01.053261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.052993    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jhfp\" (UniqueName: \"kubernetes.io/projected/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-kube-api-access-7jhfp\") pod \"maas-api-key-cleanup-29889930-kwv29\" (UID: \"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"
Oct 30 21:30:01.053874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.053043    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67kc\" (UniqueName: \"kubernetes.io/projected/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-kube-api-access-l67kc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt\" (UID: \"e1225dbb-da72-4c3f-b13d-b63d71e3ec51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"
Oct 30 21:30:01.054296 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.054268    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8cf\" (UniqueName: \"kubernetes.io/projected/43a51867-ecfe-43e9-b630-b621e9b4e428-kube-api-access-ks8cf\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx\" (UID: \"43a51867-ecfe-43e9-b630-b621e9b4e428\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"
Oct 30 21:30:01.060197 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.060175    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"]
Oct 30 21:30:01.066067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.066044    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"
Oct 30 21:30:01.068976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.068947    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"
Oct 30 21:30:01.075155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.074682    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxdjb\" (UniqueName: \"kubernetes.io/projected/8ea1b87b-1562-4d1e-af32-8078f1e62468-kube-api-access-lxdjb\") pod \"maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq\" (UID: \"8ea1b87b-1562-4d1e-af32-8078f1e62468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"
Oct 30 21:30:01.075155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.074886    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d46wp\" (UniqueName: \"kubernetes.io/projected/379363eb-5d66-41cf-989c-08cf4a2008ce-kube-api-access-d46wp\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg\" (UID: \"379363eb-5d66-41cf-989c-08cf4a2008ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"
Oct 30 21:30:01.075155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.074977    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/379363eb-5d66-41cf-989c-08cf4a2008ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg\" (UID: \"379363eb-5d66-41cf-989c-08cf4a2008ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"
Oct 30 21:30:01.075155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.075102    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f43a597-0a8a-4c75-8123-d5450be93d21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg\" (UID: \"8f43a597-0a8a-4c75-8123-d5450be93d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"
Oct 30 21:30:01.075807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.075607    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gj8g\" (UniqueName: \"kubernetes.io/projected/8f43a597-0a8a-4c75-8123-d5450be93d21-kube-api-access-5gj8g\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg\" (UID: \"8f43a597-0a8a-4c75-8123-d5450be93d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"
Oct 30 21:30:01.075807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.075680    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea1b87b-1562-4d1e-af32-8078f1e62468-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq\" (UID: \"8ea1b87b-1562-4d1e-af32-8078f1e62468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"
Oct 30 21:30:01.083130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.081309    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"
Oct 30 21:30:01.151052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.151017    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"]
Oct 30 21:30:01.151221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.151086    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"]
Oct 30 21:30:01.163868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.156671    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"]
Oct 30 21:30:01.177580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.176924    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d46wp\" (UniqueName: \"kubernetes.io/projected/379363eb-5d66-41cf-989c-08cf4a2008ce-kube-api-access-d46wp\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg\" (UID: \"379363eb-5d66-41cf-989c-08cf4a2008ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"
Oct 30 21:30:01.177580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.177031    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/379363eb-5d66-41cf-989c-08cf4a2008ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg\" (UID: \"379363eb-5d66-41cf-989c-08cf4a2008ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"
Oct 30 21:30:01.177580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.177182    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f43a597-0a8a-4c75-8123-d5450be93d21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg\" (UID: \"8f43a597-0a8a-4c75-8123-d5450be93d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"
Oct 30 21:30:01.177580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.177258    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68adc98-00aa-48da-8690-4ce89ca58a57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv\" (UID: \"c68adc98-00aa-48da-8690-4ce89ca58a57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"
Oct 30 21:30:01.177580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.177305    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17420e84-c277-42b9-99d8-c139e86395df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh\" (UID: \"17420e84-c277-42b9-99d8-c139e86395df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"
Oct 30 21:30:01.177580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.177450    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gj8g\" (UniqueName: \"kubernetes.io/projected/8f43a597-0a8a-4c75-8123-d5450be93d21-kube-api-access-5gj8g\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg\" (UID: \"8f43a597-0a8a-4c75-8123-d5450be93d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"
Oct 30 21:30:01.177580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.177527    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea1b87b-1562-4d1e-af32-8078f1e62468-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq\" (UID: \"8ea1b87b-1562-4d1e-af32-8078f1e62468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"
Oct 30 21:30:01.177580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.177557    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b657\" (UniqueName: \"kubernetes.io/projected/c68adc98-00aa-48da-8690-4ce89ca58a57-kube-api-access-7b657\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv\" (UID: \"c68adc98-00aa-48da-8690-4ce89ca58a57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"
Oct 30 21:30:01.177969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.177611    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxdjb\" (UniqueName: \"kubernetes.io/projected/8ea1b87b-1562-4d1e-af32-8078f1e62468-kube-api-access-lxdjb\") pod \"maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq\" (UID: \"8ea1b87b-1562-4d1e-af32-8078f1e62468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"
Oct 30 21:30:01.177969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.177844    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfn9\" (UniqueName: \"kubernetes.io/projected/17420e84-c277-42b9-99d8-c139e86395df-kube-api-access-4sfn9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh\" (UID: \"17420e84-c277-42b9-99d8-c139e86395df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"
Oct 30 21:30:01.181061 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.179652    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea1b87b-1562-4d1e-af32-8078f1e62468-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq\" (UID: \"8ea1b87b-1562-4d1e-af32-8078f1e62468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"
Oct 30 21:30:01.181061 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.180182    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/379363eb-5d66-41cf-989c-08cf4a2008ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg\" (UID: \"379363eb-5d66-41cf-989c-08cf4a2008ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"
Oct 30 21:30:01.181061 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.180542    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"]
Oct 30 21:30:01.181061 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.180851    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f43a597-0a8a-4c75-8123-d5450be93d21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg\" (UID: \"8f43a597-0a8a-4c75-8123-d5450be93d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"
Oct 30 21:30:01.318470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.251238    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"]
Oct 30 21:30:01.318470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.277946    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"
Oct 30 21:30:01.318470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.278727    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"
Oct 30 21:30:01.318470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.282796    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b657\" (UniqueName: \"kubernetes.io/projected/c68adc98-00aa-48da-8690-4ce89ca58a57-kube-api-access-7b657\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv\" (UID: \"c68adc98-00aa-48da-8690-4ce89ca58a57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"
Oct 30 21:30:01.318470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.282953    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfn9\" (UniqueName: \"kubernetes.io/projected/17420e84-c277-42b9-99d8-c139e86395df-kube-api-access-4sfn9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh\" (UID: \"17420e84-c277-42b9-99d8-c139e86395df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"
Oct 30 21:30:01.318470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.283284    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68adc98-00aa-48da-8690-4ce89ca58a57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv\" (UID: \"c68adc98-00aa-48da-8690-4ce89ca58a57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"
Oct 30 21:30:01.318470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.283344    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17420e84-c277-42b9-99d8-c139e86395df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh\" (UID: \"17420e84-c277-42b9-99d8-c139e86395df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"
Oct 30 21:30:01.318470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.285204    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17420e84-c277-42b9-99d8-c139e86395df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh\" (UID: \"17420e84-c277-42b9-99d8-c139e86395df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"
Oct 30 21:30:01.318470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.291119    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"
Oct 30 21:30:01.318470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.313784    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d46wp\" (UniqueName: \"kubernetes.io/projected/379363eb-5d66-41cf-989c-08cf4a2008ce-kube-api-access-d46wp\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg\" (UID: \"379363eb-5d66-41cf-989c-08cf4a2008ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"
Oct 30 21:30:01.480933 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.480907    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68adc98-00aa-48da-8690-4ce89ca58a57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv\" (UID: \"c68adc98-00aa-48da-8690-4ce89ca58a57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"
Oct 30 21:30:01.481408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.481260    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b657\" (UniqueName: \"kubernetes.io/projected/c68adc98-00aa-48da-8690-4ce89ca58a57-kube-api-access-7b657\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv\" (UID: \"c68adc98-00aa-48da-8690-4ce89ca58a57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"
Oct 30 21:30:01.485016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.484983    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"
Oct 30 21:30:01.541430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.496476    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxdjb\" (UniqueName: \"kubernetes.io/projected/8ea1b87b-1562-4d1e-af32-8078f1e62468-kube-api-access-lxdjb\") pod \"maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq\" (UID: \"8ea1b87b-1562-4d1e-af32-8078f1e62468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"
Oct 30 21:30:01.541430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.533347    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"]
Oct 30 21:30:01.543499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.542070    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gj8g\" (UniqueName: \"kubernetes.io/projected/8f43a597-0a8a-4c75-8123-d5450be93d21-kube-api-access-5gj8g\") pod \"maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg\" (UID: \"8f43a597-0a8a-4c75-8123-d5450be93d21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"
Oct 30 21:30:01.544131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.544077    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfn9\" (UniqueName: \"kubernetes.io/projected/17420e84-c277-42b9-99d8-c139e86395df-kube-api-access-4sfn9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh\" (UID: \"17420e84-c277-42b9-99d8-c139e86395df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"
Oct 30 21:30:01.546096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.546023    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"]
Oct 30 21:30:01.563347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.563123    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"
Oct 30 21:30:01.565270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.565234    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"]
Oct 30 21:30:01.757944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.757914    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"
Oct 30 21:30:01.781907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.781693    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"
Oct 30 21:30:01.848702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.846755    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"
Oct 30 21:30:01.957237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.957198    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"]
Oct 30 21:30:01.957379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.957283    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"]
Oct 30 21:30:01.965885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:01.965517    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"]
Oct 30 21:30:02.043361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:02.043302    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v" event={"ID":"fc27f4e0-65dd-4971-abee-816ac50c9869","Type":"ContainerStarted","Data":"c8be3d8116063825c18e48a86e31bb8338afba51320625c3b3cc6d0828736ff7"}
Oct 30 21:30:02.046515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:02.045446    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct" event={"ID":"9fbca41f-798b-4b86-b103-47868642c624","Type":"ContainerStarted","Data":"edc8f169412983e31fe7ed85208960c918bbc9ceca363eb7cb834c1b05e5a45f"}
Oct 30 21:30:02.062890 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:30:02.062844    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778db27c_4f22_4317_9753_b84985424b0b.slice/crio-41fafc8a4d0421c6418c75ddbd18fbc18f6cb71c638019ef94e9d0444edf9fd3 WatchSource:0}: Error finding container 41fafc8a4d0421c6418c75ddbd18fbc18f6cb71c638019ef94e9d0444edf9fd3: Status 404 returned error can't find the container with id 41fafc8a4d0421c6418c75ddbd18fbc18f6cb71c638019ef94e9d0444edf9fd3
Oct 30 21:30:02.661869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:02.661773    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"]
Oct 30 21:30:02.678842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:02.678775    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"]
Oct 30 21:30:02.740202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:02.740052    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"]
Oct 30 21:30:03.063977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.063883    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz" event={"ID":"778db27c-4f22-4317-9753-b84985424b0b","Type":"ContainerStarted","Data":"41fafc8a4d0421c6418c75ddbd18fbc18f6cb71c638019ef94e9d0444edf9fd3"}
Oct 30 21:30:03.065746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.065721    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx" event={"ID":"43a51867-ecfe-43e9-b630-b621e9b4e428","Type":"ContainerStarted","Data":"0d51eb87e23378d5a96597dc8d30e93c97614083569599c0c21b014a129ca06e"}
Oct 30 21:30:03.067476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.067445    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29" event={"ID":"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418","Type":"ContainerStarted","Data":"b4c5549fdd79ad2f91f5e56ee51770c4e09c30e10cf226368de698072295ccc0"}
Oct 30 21:30:03.154452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.143623    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v" event={"ID":"fc27f4e0-65dd-4971-abee-816ac50c9869","Type":"ContainerStarted","Data":"0122f3f924637be87bf15ae2e682794f336f5d8a67c234be5760b0a1ba123375"}
Oct 30 21:30:03.156246 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.155805    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct" event={"ID":"9fbca41f-798b-4b86-b103-47868642c624","Type":"ContainerStarted","Data":"4658ae47d20df878febaf24a5a9f418a4b04e656ee348379512a768a7913f232"}
Oct 30 21:30:03.158982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.158891    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v" podStartSLOduration=3.158872327 podStartE2EDuration="3.158872327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.115123021 +0000 UTC m=+1716.161682167" watchObservedRunningTime="2026-10-30 21:30:03.158872327 +0000 UTC m=+1716.205431474"
Oct 30 21:30:03.160526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.160359    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65" event={"ID":"1a6c38be-7c77-435c-a127-3d3fd054574e","Type":"ContainerStarted","Data":"6333b4b2fe7ae4ac6588c8cb4924690261b4ceece0e0143214c319ac2e25815c"}
Oct 30 21:30:03.169695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.169585    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"]
Oct 30 21:30:03.174302 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:30:03.174268    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379363eb_5d66_41cf_989c_08cf4a2008ce.slice/crio-bd9d577f6943004a5a41bb9a4a04f6a0e4485b6445fb84f6bd7dd293d0f885dc WatchSource:0}: Error finding container bd9d577f6943004a5a41bb9a4a04f6a0e4485b6445fb84f6bd7dd293d0f885dc: Status 404 returned error can't find the container with id bd9d577f6943004a5a41bb9a4a04f6a0e4485b6445fb84f6bd7dd293d0f885dc
Oct 30 21:30:03.174526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.174462    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"]
Oct 30 21:30:03.174874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.174831    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct" podStartSLOduration=3.174819387 podStartE2EDuration="3.174819387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.143129828 +0000 UTC m=+1716.189688988" watchObservedRunningTime="2026-10-30 21:30:03.174819387 +0000 UTC m=+1716.221378533"
Oct 30 21:30:03.254795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.254526    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"]
Oct 30 21:30:03.260490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.260461    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"]
Oct 30 21:30:03.266799 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.266485    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"]
Oct 30 21:30:03.267532 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:30:03.267512    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea1b87b_1562_4d1e_af32_8078f1e62468.slice/crio-ebe27b83a17fbef729e27d403ecae7f062e46d66902e253a724f6e51ee142e3c WatchSource:0}: Error finding container ebe27b83a17fbef729e27d403ecae7f062e46d66902e253a724f6e51ee142e3c: Status 404 returned error can't find the container with id ebe27b83a17fbef729e27d403ecae7f062e46d66902e253a724f6e51ee142e3c
Oct 30 21:30:03.352446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:03.352299    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"]
Oct 30 21:30:04.171064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.171029    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv" event={"ID":"c68adc98-00aa-48da-8690-4ce89ca58a57","Type":"ContainerStarted","Data":"727abd2997156dbb2f4d865a4380aaae0e39e49c2ffd8c40f7593cd396206110"}
Oct 30 21:30:04.171493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.171099    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv" event={"ID":"c68adc98-00aa-48da-8690-4ce89ca58a57","Type":"ContainerStarted","Data":"04e4b7877ba227831c747d0361ac774ae33b6bd69be23d415814b49b6b318f01"}
Oct 30 21:30:04.172717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.172695    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg" event={"ID":"8f43a597-0a8a-4c75-8123-d5450be93d21","Type":"ContainerStarted","Data":"8f19bcb17e5b898913a4b57ed67c639428994a0536803c954f6dc640a6117f3d"}
Oct 30 21:30:04.174076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.174055    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh" event={"ID":"17420e84-c277-42b9-99d8-c139e86395df","Type":"ContainerStarted","Data":"7787f75ad68c6bd061d89f4a153f0147b5d47b793ed561188841376c08d6bdc8"}
Oct 30 21:30:04.175813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.175583    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg" event={"ID":"379363eb-5d66-41cf-989c-08cf4a2008ce","Type":"ContainerStarted","Data":"2fdf856458c0724e9c0bd658423a198137e1701478a4242f52d081d17a94bfab"}
Oct 30 21:30:04.175813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.175621    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg" event={"ID":"379363eb-5d66-41cf-989c-08cf4a2008ce","Type":"ContainerStarted","Data":"bd9d577f6943004a5a41bb9a4a04f6a0e4485b6445fb84f6bd7dd293d0f885dc"}
Oct 30 21:30:04.177037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.177017    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65" event={"ID":"1a6c38be-7c77-435c-a127-3d3fd054574e","Type":"ContainerStarted","Data":"d52f60a25f2f9924aeba1113cd45363d597188d0ebfc54d6949b67ccec710177"}
Oct 30 21:30:04.178571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.178552    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz" event={"ID":"778db27c-4f22-4317-9753-b84985424b0b","Type":"ContainerStarted","Data":"8c365d53cbfde71d81d872dd2e9913f83184309dfd06e7617e91d3740aa5357a"}
Oct 30 21:30:04.179982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.179960    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx" event={"ID":"43a51867-ecfe-43e9-b630-b621e9b4e428","Type":"ContainerStarted","Data":"12f319cc9106e520e5cd1d8f8b17b727249f8119e1c75b519caf5213e5019a97"}
Oct 30 21:30:04.181252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.181217    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq" event={"ID":"8ea1b87b-1562-4d1e-af32-8078f1e62468","Type":"ContainerStarted","Data":"ebe27b83a17fbef729e27d403ecae7f062e46d66902e253a724f6e51ee142e3c"}
Oct 30 21:30:04.182843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.182823    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29" event={"ID":"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418","Type":"ContainerStarted","Data":"62e2fd55bc7edabfaeeee135e26d4fa2aad6eb29dce015df4fed283213476ac7"}
Oct 30 21:30:04.184721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.184377    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt" event={"ID":"e1225dbb-da72-4c3f-b13d-b63d71e3ec51","Type":"ContainerStarted","Data":"928e1514ee5149b0691df40acb56a649ae3471ee9959495dfac4ed36fed5182b"}
Oct 30 21:30:04.184721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.184444    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt" event={"ID":"e1225dbb-da72-4c3f-b13d-b63d71e3ec51","Type":"ContainerStarted","Data":"b295adce2deccb70b86e5f95e7837876da7e77745c839fdee30924ad504cf1ec"}
Oct 30 21:30:04.203424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.197494    4918 generic.go:356] "Generic (PLEG): container finished" podID="fc27f4e0-65dd-4971-abee-816ac50c9869" containerID="0122f3f924637be87bf15ae2e682794f336f5d8a67c234be5760b0a1ba123375" exitCode=60
Oct 30 21:30:04.203424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.197591    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v" event={"ID":"fc27f4e0-65dd-4971-abee-816ac50c9869","Type":"ContainerDied","Data":"0122f3f924637be87bf15ae2e682794f336f5d8a67c234be5760b0a1ba123375"}
Oct 30 21:30:04.203424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.198085    4918 scope.go:122] "RemoveContainer" containerID="0122f3f924637be87bf15ae2e682794f336f5d8a67c234be5760b0a1ba123375"
Oct 30 21:30:04.218345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.218286    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv" podStartSLOduration=4.218262175 podStartE2EDuration="4.218262175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.200628704 +0000 UTC m=+1717.247187886" watchObservedRunningTime="2026-10-30 21:30:04.218262175 +0000 UTC m=+1717.264821322"
Oct 30 21:30:04.258119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.258058    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt" podStartSLOduration=4.258039902 podStartE2EDuration="4.258039902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.243144959 +0000 UTC m=+1717.289704133" watchObservedRunningTime="2026-10-30 21:30:04.258039902 +0000 UTC m=+1717.304599047"
Oct 30 21:30:04.259700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.259278    4918 generic.go:356] "Generic (PLEG): container finished" podID="9fbca41f-798b-4b86-b103-47868642c624" containerID="4658ae47d20df878febaf24a5a9f418a4b04e656ee348379512a768a7913f232" exitCode=60
Oct 30 21:30:04.259700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.259345    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct" event={"ID":"9fbca41f-798b-4b86-b103-47868642c624","Type":"ContainerDied","Data":"4658ae47d20df878febaf24a5a9f418a4b04e656ee348379512a768a7913f232"}
Oct 30 21:30:04.259940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.259821    4918 scope.go:122] "RemoveContainer" containerID="4658ae47d20df878febaf24a5a9f418a4b04e656ee348379512a768a7913f232"
Oct 30 21:30:04.270533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.270482    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg" podStartSLOduration=4.2704658779999995 podStartE2EDuration="4.270465878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.268227983 +0000 UTC m=+1717.314787142" watchObservedRunningTime="2026-10-30 21:30:04.270465878 +0000 UTC m=+1717.317025024"
Oct 30 21:30:04.371208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.370546    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65" podStartSLOduration=4.3705273909999995 podStartE2EDuration="4.370527391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.369527889 +0000 UTC m=+1717.416087061" watchObservedRunningTime="2026-10-30 21:30:04.370527391 +0000 UTC m=+1717.417086538"
Oct 30 21:30:04.449657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.448811    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz" podStartSLOduration=4.448787525 podStartE2EDuration="4.448787525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.417530189 +0000 UTC m=+1717.464089370" watchObservedRunningTime="2026-10-30 21:30:04.448787525 +0000 UTC m=+1717.495346674"
Oct 30 21:30:04.516693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.507287    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29" podStartSLOduration=4.507262127 podStartE2EDuration="4.507262127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.45295031 +0000 UTC m=+1717.499509467" watchObservedRunningTime="2026-10-30 21:30:04.507262127 +0000 UTC m=+1717.553821273"
Oct 30 21:30:04.711424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:04.686503    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx" podStartSLOduration=4.686484187 podStartE2EDuration="4.686484187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.571066691 +0000 UTC m=+1717.617625837" watchObservedRunningTime="2026-10-30 21:30:04.686484187 +0000 UTC m=+1717.733043337"
Oct 30 21:30:05.278447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.274248    4918 generic.go:356] "Generic (PLEG): container finished" podID="1a6c38be-7c77-435c-a127-3d3fd054574e" containerID="d52f60a25f2f9924aeba1113cd45363d597188d0ebfc54d6949b67ccec710177" exitCode=60
Oct 30 21:30:05.278447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.274448    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65" event={"ID":"1a6c38be-7c77-435c-a127-3d3fd054574e","Type":"ContainerDied","Data":"d52f60a25f2f9924aeba1113cd45363d597188d0ebfc54d6949b67ccec710177"}
Oct 30 21:30:05.295632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.281141    4918 generic.go:356] "Generic (PLEG): container finished" podID="778db27c-4f22-4317-9753-b84985424b0b" containerID="8c365d53cbfde71d81d872dd2e9913f83184309dfd06e7617e91d3740aa5357a" exitCode=60
Oct 30 21:30:05.295632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.281250    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz" event={"ID":"778db27c-4f22-4317-9753-b84985424b0b","Type":"ContainerDied","Data":"8c365d53cbfde71d81d872dd2e9913f83184309dfd06e7617e91d3740aa5357a"}
Oct 30 21:30:05.295632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.282649    4918 scope.go:122] "RemoveContainer" containerID="d52f60a25f2f9924aeba1113cd45363d597188d0ebfc54d6949b67ccec710177"
Oct 30 21:30:05.295632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.284510    4918 scope.go:122] "RemoveContainer" containerID="8c365d53cbfde71d81d872dd2e9913f83184309dfd06e7617e91d3740aa5357a"
Oct 30 21:30:05.368424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.361770    4918 generic.go:356] "Generic (PLEG): container finished" podID="43a51867-ecfe-43e9-b630-b621e9b4e428" containerID="12f319cc9106e520e5cd1d8f8b17b727249f8119e1c75b519caf5213e5019a97" exitCode=60
Oct 30 21:30:05.368424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.361892    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx" event={"ID":"43a51867-ecfe-43e9-b630-b621e9b4e428","Type":"ContainerDied","Data":"12f319cc9106e520e5cd1d8f8b17b727249f8119e1c75b519caf5213e5019a97"}
Oct 30 21:30:05.368424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.362516    4918 scope.go:122] "RemoveContainer" containerID="12f319cc9106e520e5cd1d8f8b17b727249f8119e1c75b519caf5213e5019a97"
Oct 30 21:30:05.447415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.445238    4918 generic.go:356] "Generic (PLEG): container finished" podID="8ea1b87b-1562-4d1e-af32-8078f1e62468" containerID="a04752f09b1649f59194bea763dfdffdb25fd80f4bb7208ba097cba5b188c352" exitCode=60
Oct 30 21:30:05.447415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.445446    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq" event={"ID":"8ea1b87b-1562-4d1e-af32-8078f1e62468","Type":"ContainerDied","Data":"a04752f09b1649f59194bea763dfdffdb25fd80f4bb7208ba097cba5b188c352"}
Oct 30 21:30:05.447415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.446225    4918 scope.go:122] "RemoveContainer" containerID="a04752f09b1649f59194bea763dfdffdb25fd80f4bb7208ba097cba5b188c352"
Oct 30 21:30:05.470685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.463258    4918 generic.go:356] "Generic (PLEG): container finished" podID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" containerID="62e2fd55bc7edabfaeeee135e26d4fa2aad6eb29dce015df4fed283213476ac7" exitCode=60
Oct 30 21:30:05.470685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.463371    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29" event={"ID":"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418","Type":"ContainerDied","Data":"62e2fd55bc7edabfaeeee135e26d4fa2aad6eb29dce015df4fed283213476ac7"}
Oct 30 21:30:05.470685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.463894    4918 scope.go:122] "RemoveContainer" containerID="62e2fd55bc7edabfaeeee135e26d4fa2aad6eb29dce015df4fed283213476ac7"
Oct 30 21:30:05.480323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.480297    4918 generic.go:356] "Generic (PLEG): container finished" podID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" containerID="928e1514ee5149b0691df40acb56a649ae3471ee9959495dfac4ed36fed5182b" exitCode=60
Oct 30 21:30:05.480502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.480473    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt" event={"ID":"e1225dbb-da72-4c3f-b13d-b63d71e3ec51","Type":"ContainerDied","Data":"928e1514ee5149b0691df40acb56a649ae3471ee9959495dfac4ed36fed5182b"}
Oct 30 21:30:05.481032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.480842    4918 scope.go:122] "RemoveContainer" containerID="928e1514ee5149b0691df40acb56a649ae3471ee9959495dfac4ed36fed5182b"
Oct 30 21:30:05.483722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.483695    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v" event={"ID":"fc27f4e0-65dd-4971-abee-816ac50c9869","Type":"ContainerStarted","Data":"88f71f65f7ee15edbd802de20ab7b95278bb2e3d829afcffe1736f8f12d65d24"}
Oct 30 21:30:05.545054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.545024    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct" event={"ID":"9fbca41f-798b-4b86-b103-47868642c624","Type":"ContainerStarted","Data":"4ad71ebddab5db71ef28040b2d1f51f6e4fb0a5ded493788067f594fad4fa246"}
Oct 30 21:30:05.555149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.555123    4918 generic.go:356] "Generic (PLEG): container finished" podID="c68adc98-00aa-48da-8690-4ce89ca58a57" containerID="727abd2997156dbb2f4d865a4380aaae0e39e49c2ffd8c40f7593cd396206110" exitCode=60
Oct 30 21:30:05.555756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.555737    4918 scope.go:122] "RemoveContainer" containerID="727abd2997156dbb2f4d865a4380aaae0e39e49c2ffd8c40f7593cd396206110"
Oct 30 21:30:05.556728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.556297    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv" event={"ID":"c68adc98-00aa-48da-8690-4ce89ca58a57","Type":"ContainerDied","Data":"727abd2997156dbb2f4d865a4380aaae0e39e49c2ffd8c40f7593cd396206110"}
Oct 30 21:30:05.560457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.560431    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg" event={"ID":"8f43a597-0a8a-4c75-8123-d5450be93d21","Type":"ContainerStarted","Data":"fcc509d273cd704ca007d103f6ca8e68b567d4de407c5ab338ea87be8772665d"}
Oct 30 21:30:05.562275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.561194    4918 scope.go:122] "RemoveContainer" containerID="fcc509d273cd704ca007d103f6ca8e68b567d4de407c5ab338ea87be8772665d"
Oct 30 21:30:05.564161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.564137    4918 generic.go:356] "Generic (PLEG): container finished" podID="17420e84-c277-42b9-99d8-c139e86395df" containerID="bfc446e634ed221d8e2956be1622c3e7e4e8a07fef93994e772fbd79613a3719" exitCode=60
Oct 30 21:30:05.564285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.564267    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh" event={"ID":"17420e84-c277-42b9-99d8-c139e86395df","Type":"ContainerDied","Data":"bfc446e634ed221d8e2956be1622c3e7e4e8a07fef93994e772fbd79613a3719"}
Oct 30 21:30:05.564477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.564457    4918 scope.go:122] "RemoveContainer" containerID="bfc446e634ed221d8e2956be1622c3e7e4e8a07fef93994e772fbd79613a3719"
Oct 30 21:30:05.566814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.566788    4918 generic.go:356] "Generic (PLEG): container finished" podID="379363eb-5d66-41cf-989c-08cf4a2008ce" containerID="2fdf856458c0724e9c0bd658423a198137e1701478a4242f52d081d17a94bfab" exitCode=60
Oct 30 21:30:05.566918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.566846    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg" event={"ID":"379363eb-5d66-41cf-989c-08cf4a2008ce","Type":"ContainerDied","Data":"2fdf856458c0724e9c0bd658423a198137e1701478a4242f52d081d17a94bfab"}
Oct 30 21:30:05.567080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:05.567064    4918 scope.go:122] "RemoveContainer" containerID="2fdf856458c0724e9c0bd658423a198137e1701478a4242f52d081d17a94bfab"
Oct 30 21:30:06.579329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.579293    4918 generic.go:356] "Generic (PLEG): container finished" podID="fc27f4e0-65dd-4971-abee-816ac50c9869" containerID="88f71f65f7ee15edbd802de20ab7b95278bb2e3d829afcffe1736f8f12d65d24" exitCode=60
Oct 30 21:30:06.579883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.579703    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v" event={"ID":"fc27f4e0-65dd-4971-abee-816ac50c9869","Type":"ContainerDied","Data":"88f71f65f7ee15edbd802de20ab7b95278bb2e3d829afcffe1736f8f12d65d24"}
Oct 30 21:30:06.579883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.579802    4918 scope.go:122] "RemoveContainer" containerID="0122f3f924637be87bf15ae2e682794f336f5d8a67c234be5760b0a1ba123375"
Oct 30 21:30:06.580112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.580093    4918 scope.go:122] "RemoveContainer" containerID="88f71f65f7ee15edbd802de20ab7b95278bb2e3d829afcffe1736f8f12d65d24"
Oct 30 21:30:06.580612 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:06.580589    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v_odh-ai-gateway-infra(fc27f4e0-65dd-4971-abee-816ac50c9869)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869"
Oct 30 21:30:06.582794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.582769    4918 generic.go:356] "Generic (PLEG): container finished" podID="9fbca41f-798b-4b86-b103-47868642c624" containerID="4ad71ebddab5db71ef28040b2d1f51f6e4fb0a5ded493788067f594fad4fa246" exitCode=60
Oct 30 21:30:06.583240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.583227    4918 scope.go:122] "RemoveContainer" containerID="4ad71ebddab5db71ef28040b2d1f51f6e4fb0a5ded493788067f594fad4fa246"
Oct 30 21:30:06.583545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.583500    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct" event={"ID":"9fbca41f-798b-4b86-b103-47868642c624","Type":"ContainerDied","Data":"4ad71ebddab5db71ef28040b2d1f51f6e4fb0a5ded493788067f594fad4fa246"}
Oct 30 21:30:06.583640 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:06.583552    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct_odh-ai-gateway-infra(9fbca41f-798b-4b86-b103-47868642c624)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct" podUID="9fbca41f-798b-4b86-b103-47868642c624"
Oct 30 21:30:06.585359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.585335    4918 generic.go:356] "Generic (PLEG): container finished" podID="8f43a597-0a8a-4c75-8123-d5450be93d21" containerID="fcc509d273cd704ca007d103f6ca8e68b567d4de407c5ab338ea87be8772665d" exitCode=60
Oct 30 21:30:06.596442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.589589    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg" event={"ID":"8f43a597-0a8a-4c75-8123-d5450be93d21","Type":"ContainerDied","Data":"fcc509d273cd704ca007d103f6ca8e68b567d4de407c5ab338ea87be8772665d"}
Oct 30 21:30:06.608837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.608790    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65" event={"ID":"1a6c38be-7c77-435c-a127-3d3fd054574e","Type":"ContainerStarted","Data":"542dd579d4ac3bedc5cb32230646b2979b6013577896bb77de180b22248a8ca9"}
Oct 30 21:30:06.623630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.623533    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz" event={"ID":"778db27c-4f22-4317-9753-b84985424b0b","Type":"ContainerStarted","Data":"0cb2fd82357ec9ec421baf309ddb3a928fc5fbdc0cc8f2fea183c03880a264f6"}
Oct 30 21:30:06.640290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.640251    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx" event={"ID":"43a51867-ecfe-43e9-b630-b621e9b4e428","Type":"ContainerStarted","Data":"c91db332f1ab74267875d56766f070679bee757cea3bb37ec5892f7c71a22c06"}
Oct 30 21:30:06.651112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.651083    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq" event={"ID":"8ea1b87b-1562-4d1e-af32-8078f1e62468","Type":"ContainerStarted","Data":"95657a11862a7e5069f5846cf96c8aaeffe957270f7a0e44ccb7726a7997b85d"}
Oct 30 21:30:06.656516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.656446    4918 scope.go:122] "RemoveContainer" containerID="4658ae47d20df878febaf24a5a9f418a4b04e656ee348379512a768a7913f232"
Oct 30 21:30:06.760721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:06.760658    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq" podStartSLOduration=6.760639539 podStartE2EDuration="6.760639539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:06.752329971 +0000 UTC m=+1719.798889116" watchObservedRunningTime="2026-10-30 21:30:06.760639539 +0000 UTC m=+1719.807198684"
Oct 30 21:30:07.670530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.670427    4918 generic.go:356] "Generic (PLEG): container finished" podID="778db27c-4f22-4317-9753-b84985424b0b" containerID="0cb2fd82357ec9ec421baf309ddb3a928fc5fbdc0cc8f2fea183c03880a264f6" exitCode=60
Oct 30 21:30:07.671084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.670525    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz" event={"ID":"778db27c-4f22-4317-9753-b84985424b0b","Type":"ContainerDied","Data":"0cb2fd82357ec9ec421baf309ddb3a928fc5fbdc0cc8f2fea183c03880a264f6"}
Oct 30 21:30:07.671084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.670592    4918 scope.go:122] "RemoveContainer" containerID="8c365d53cbfde71d81d872dd2e9913f83184309dfd06e7617e91d3740aa5357a"
Oct 30 21:30:07.671084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.671049    4918 scope.go:122] "RemoveContainer" containerID="0cb2fd82357ec9ec421baf309ddb3a928fc5fbdc0cc8f2fea183c03880a264f6"
Oct 30 21:30:07.671742 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:07.671549    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz_odh-ai-gateway-infra(778db27c-4f22-4317-9753-b84985424b0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz" podUID="778db27c-4f22-4317-9753-b84985424b0b"
Oct 30 21:30:07.675605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.675584    4918 generic.go:356] "Generic (PLEG): container finished" podID="43a51867-ecfe-43e9-b630-b621e9b4e428" containerID="c91db332f1ab74267875d56766f070679bee757cea3bb37ec5892f7c71a22c06" exitCode=60
Oct 30 21:30:07.675697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.675672    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx" event={"ID":"43a51867-ecfe-43e9-b630-b621e9b4e428","Type":"ContainerDied","Data":"c91db332f1ab74267875d56766f070679bee757cea3bb37ec5892f7c71a22c06"}
Oct 30 21:30:07.676126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.676105    4918 scope.go:122] "RemoveContainer" containerID="c91db332f1ab74267875d56766f070679bee757cea3bb37ec5892f7c71a22c06"
Oct 30 21:30:07.676542 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:07.676513    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx_odh-ai-gateway-infra(43a51867-ecfe-43e9-b630-b621e9b4e428)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428"
Oct 30 21:30:07.677903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.677881    4918 generic.go:356] "Generic (PLEG): container finished" podID="8ea1b87b-1562-4d1e-af32-8078f1e62468" containerID="95657a11862a7e5069f5846cf96c8aaeffe957270f7a0e44ccb7726a7997b85d" exitCode=60
Oct 30 21:30:07.678035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.678019    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq" event={"ID":"8ea1b87b-1562-4d1e-af32-8078f1e62468","Type":"ContainerDied","Data":"95657a11862a7e5069f5846cf96c8aaeffe957270f7a0e44ccb7726a7997b85d"}
Oct 30 21:30:07.678383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.678365    4918 scope.go:122] "RemoveContainer" containerID="95657a11862a7e5069f5846cf96c8aaeffe957270f7a0e44ccb7726a7997b85d"
Oct 30 21:30:07.678770 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:07.678746    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq_odh-ai-gateway-infra(8ea1b87b-1562-4d1e-af32-8078f1e62468)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468"
Oct 30 21:30:07.680581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.680559    4918 generic.go:356] "Generic (PLEG): container finished" podID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" containerID="61c395261e0962493d0bab12694af5d243e44cf0172a9f1de42e97f56d10691a" exitCode=60
Oct 30 21:30:07.680662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.680636    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29" event={"ID":"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418","Type":"ContainerDied","Data":"61c395261e0962493d0bab12694af5d243e44cf0172a9f1de42e97f56d10691a"}
Oct 30 21:30:07.680965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.680947    4918 scope.go:122] "RemoveContainer" containerID="61c395261e0962493d0bab12694af5d243e44cf0172a9f1de42e97f56d10691a"
Oct 30 21:30:07.681242 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:07.681225    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-kwv29_odh-ai-gateway-infra(0a9f2c2a-8aea-4ecf-941b-de85d3bc6418)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418"
Oct 30 21:30:07.683430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.683361    4918 generic.go:356] "Generic (PLEG): container finished" podID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" containerID="0dfaea4a4507154b74bc1b6c7bbdd21bc6617ff3ce56bc3d606010ddff58f61c" exitCode=60
Oct 30 21:30:07.683529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.683510    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt" event={"ID":"e1225dbb-da72-4c3f-b13d-b63d71e3ec51","Type":"ContainerDied","Data":"0dfaea4a4507154b74bc1b6c7bbdd21bc6617ff3ce56bc3d606010ddff58f61c"}
Oct 30 21:30:07.683784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.683769    4918 scope.go:122] "RemoveContainer" containerID="0dfaea4a4507154b74bc1b6c7bbdd21bc6617ff3ce56bc3d606010ddff58f61c"
Oct 30 21:30:07.684056 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:07.684037    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt_odh-ai-gateway-infra(e1225dbb-da72-4c3f-b13d-b63d71e3ec51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51"
Oct 30 21:30:07.707452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.691475    4918 scope.go:122] "RemoveContainer" containerID="88f71f65f7ee15edbd802de20ab7b95278bb2e3d829afcffe1736f8f12d65d24"
Oct 30 21:30:07.707452 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:07.691772    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v_odh-ai-gateway-infra(fc27f4e0-65dd-4971-abee-816ac50c9869)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869"
Oct 30 21:30:07.707452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.699306    4918 scope.go:122] "RemoveContainer" containerID="12f319cc9106e520e5cd1d8f8b17b727249f8119e1c75b519caf5213e5019a97"
Oct 30 21:30:07.707452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.700737    4918 scope.go:122] "RemoveContainer" containerID="4ad71ebddab5db71ef28040b2d1f51f6e4fb0a5ded493788067f594fad4fa246"
Oct 30 21:30:07.707452 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:07.701012    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct_odh-ai-gateway-infra(9fbca41f-798b-4b86-b103-47868642c624)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct" podUID="9fbca41f-798b-4b86-b103-47868642c624"
Oct 30 21:30:07.707452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.703647    4918 generic.go:356] "Generic (PLEG): container finished" podID="c68adc98-00aa-48da-8690-4ce89ca58a57" containerID="2b496008ba54bbaa2990a30bbb1d5cd2ddbaa423451188874971cd7a2517780b" exitCode=60
Oct 30 21:30:07.707452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.703718    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv" event={"ID":"c68adc98-00aa-48da-8690-4ce89ca58a57","Type":"ContainerDied","Data":"2b496008ba54bbaa2990a30bbb1d5cd2ddbaa423451188874971cd7a2517780b"}
Oct 30 21:30:07.707452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.703956    4918 scope.go:122] "RemoveContainer" containerID="2b496008ba54bbaa2990a30bbb1d5cd2ddbaa423451188874971cd7a2517780b"
Oct 30 21:30:07.707452 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:07.704232    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv_odh-ai-gateway-infra(c68adc98-00aa-48da-8690-4ce89ca58a57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57"
Oct 30 21:30:07.707452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.707085    4918 generic.go:356] "Generic (PLEG): container finished" podID="8f43a597-0a8a-4c75-8123-d5450be93d21" containerID="3dc69d1e1ba090da09e5f4150c64d906ebb93d64f138bd6ec2915a6c38d58ac9" exitCode=60
Oct 30 21:30:07.707452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.707153    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg" event={"ID":"8f43a597-0a8a-4c75-8123-d5450be93d21","Type":"ContainerDied","Data":"3dc69d1e1ba090da09e5f4150c64d906ebb93d64f138bd6ec2915a6c38d58ac9"}
Oct 30 21:30:07.708366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.707530    4918 scope.go:122] "RemoveContainer" containerID="3dc69d1e1ba090da09e5f4150c64d906ebb93d64f138bd6ec2915a6c38d58ac9"
Oct 30 21:30:07.708366 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:07.707965    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg_odh-ai-gateway-infra(8f43a597-0a8a-4c75-8123-d5450be93d21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg" podUID="8f43a597-0a8a-4c75-8123-d5450be93d21"
Oct 30 21:30:07.712251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.712227    4918 generic.go:356] "Generic (PLEG): container finished" podID="17420e84-c277-42b9-99d8-c139e86395df" containerID="e2a3b95d733594f9792569173c834a8a1b9ad8f283eb9ada52d3ef22ddbb60e3" exitCode=60
Oct 30 21:30:07.712422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.712356    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh" event={"ID":"17420e84-c277-42b9-99d8-c139e86395df","Type":"ContainerDied","Data":"e2a3b95d733594f9792569173c834a8a1b9ad8f283eb9ada52d3ef22ddbb60e3"}
Oct 30 21:30:07.712671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.712639    4918 scope.go:122] "RemoveContainer" containerID="e2a3b95d733594f9792569173c834a8a1b9ad8f283eb9ada52d3ef22ddbb60e3"
Oct 30 21:30:07.712947 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:07.712921    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh_odh-ai-gateway-infra(17420e84-c277-42b9-99d8-c139e86395df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh" podUID="17420e84-c277-42b9-99d8-c139e86395df"
Oct 30 21:30:07.714742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.714718    4918 generic.go:356] "Generic (PLEG): container finished" podID="379363eb-5d66-41cf-989c-08cf4a2008ce" containerID="9be5356266ff4e8c77c851d719be544e5832e55d69786117255fa3ca0ab8cfd3" exitCode=60
Oct 30 21:30:07.714848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.714797    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg" event={"ID":"379363eb-5d66-41cf-989c-08cf4a2008ce","Type":"ContainerDied","Data":"9be5356266ff4e8c77c851d719be544e5832e55d69786117255fa3ca0ab8cfd3"}
Oct 30 21:30:07.715066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.715050    4918 scope.go:122] "RemoveContainer" containerID="9be5356266ff4e8c77c851d719be544e5832e55d69786117255fa3ca0ab8cfd3"
Oct 30 21:30:07.715345 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:07.715319    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg_odh-ai-gateway-infra(379363eb-5d66-41cf-989c-08cf4a2008ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce"
Oct 30 21:30:07.716805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.716784    4918 generic.go:356] "Generic (PLEG): container finished" podID="1a6c38be-7c77-435c-a127-3d3fd054574e" containerID="542dd579d4ac3bedc5cb32230646b2979b6013577896bb77de180b22248a8ca9" exitCode=60
Oct 30 21:30:07.716893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.716861    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65" event={"ID":"1a6c38be-7c77-435c-a127-3d3fd054574e","Type":"ContainerDied","Data":"542dd579d4ac3bedc5cb32230646b2979b6013577896bb77de180b22248a8ca9"}
Oct 30 21:30:07.717036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.717025    4918 scope.go:122] "RemoveContainer" containerID="542dd579d4ac3bedc5cb32230646b2979b6013577896bb77de180b22248a8ca9"
Oct 30 21:30:07.717242 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:07.717225    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65_odh-ai-gateway-infra(1a6c38be-7c77-435c-a127-3d3fd054574e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e"
Oct 30 21:30:07.720085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.720053    4918 scope.go:122] "RemoveContainer" containerID="a04752f09b1649f59194bea763dfdffdb25fd80f4bb7208ba097cba5b188c352"
Oct 30 21:30:07.740170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.740148    4918 scope.go:122] "RemoveContainer" containerID="62e2fd55bc7edabfaeeee135e26d4fa2aad6eb29dce015df4fed283213476ac7"
Oct 30 21:30:07.758284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.758262    4918 scope.go:122] "RemoveContainer" containerID="928e1514ee5149b0691df40acb56a649ae3471ee9959495dfac4ed36fed5182b"
Oct 30 21:30:07.775715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.775697    4918 scope.go:122] "RemoveContainer" containerID="727abd2997156dbb2f4d865a4380aaae0e39e49c2ffd8c40f7593cd396206110"
Oct 30 21:30:07.795084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.795059    4918 scope.go:122] "RemoveContainer" containerID="fcc509d273cd704ca007d103f6ca8e68b567d4de407c5ab338ea87be8772665d"
Oct 30 21:30:07.811240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.810455    4918 scope.go:122] "RemoveContainer" containerID="bfc446e634ed221d8e2956be1622c3e7e4e8a07fef93994e772fbd79613a3719"
Oct 30 21:30:07.827153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.827125    4918 scope.go:122] "RemoveContainer" containerID="2fdf856458c0724e9c0bd658423a198137e1701478a4242f52d081d17a94bfab"
Oct 30 21:30:07.846278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:07.846260    4918 scope.go:122] "RemoveContainer" containerID="d52f60a25f2f9924aeba1113cd45363d597188d0ebfc54d6949b67ccec710177"
Oct 30 21:30:08.066728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:08.066671    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"]
Oct 30 21:30:08.073630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:08.073598    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bcff5c9-jmr8b"]
Oct 30 21:30:08.731127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:08.731095    4918 scope.go:122] "RemoveContainer" containerID="61c395261e0962493d0bab12694af5d243e44cf0172a9f1de42e97f56d10691a"
Oct 30 21:30:08.731625 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:08.731596    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-kwv29_odh-ai-gateway-infra(0a9f2c2a-8aea-4ecf-941b-de85d3bc6418)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418"
Oct 30 21:30:08.734607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:08.734577    4918 scope.go:122] "RemoveContainer" containerID="0dfaea4a4507154b74bc1b6c7bbdd21bc6617ff3ce56bc3d606010ddff58f61c"
Oct 30 21:30:08.735038 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:08.735010    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt_odh-ai-gateway-infra(e1225dbb-da72-4c3f-b13d-b63d71e3ec51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51"
Oct 30 21:30:08.736607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:08.736585    4918 scope.go:122] "RemoveContainer" containerID="2b496008ba54bbaa2990a30bbb1d5cd2ddbaa423451188874971cd7a2517780b"
Oct 30 21:30:08.736959 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:08.736910    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv_odh-ai-gateway-infra(c68adc98-00aa-48da-8690-4ce89ca58a57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57"
Oct 30 21:30:08.738778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:08.738754    4918 scope.go:122] "RemoveContainer" containerID="3dc69d1e1ba090da09e5f4150c64d906ebb93d64f138bd6ec2915a6c38d58ac9"
Oct 30 21:30:08.739083 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:08.739056    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg_odh-ai-gateway-infra(8f43a597-0a8a-4c75-8123-d5450be93d21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg" podUID="8f43a597-0a8a-4c75-8123-d5450be93d21"
Oct 30 21:30:08.740595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:08.740576    4918 scope.go:122] "RemoveContainer" containerID="e2a3b95d733594f9792569173c834a8a1b9ad8f283eb9ada52d3ef22ddbb60e3"
Oct 30 21:30:08.740851 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:08.740825    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh_odh-ai-gateway-infra(17420e84-c277-42b9-99d8-c139e86395df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh" podUID="17420e84-c277-42b9-99d8-c139e86395df"
Oct 30 21:30:08.742772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:08.742753    4918 scope.go:122] "RemoveContainer" containerID="9be5356266ff4e8c77c851d719be544e5832e55d69786117255fa3ca0ab8cfd3"
Oct 30 21:30:08.742999 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:08.742981    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg_odh-ai-gateway-infra(379363eb-5d66-41cf-989c-08cf4a2008ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce"
Oct 30 21:30:08.744893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:08.744873    4918 scope.go:122] "RemoveContainer" containerID="542dd579d4ac3bedc5cb32230646b2979b6013577896bb77de180b22248a8ca9"
Oct 30 21:30:08.745181 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:08.745156    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65_odh-ai-gateway-infra(1a6c38be-7c77-435c-a127-3d3fd054574e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e"
Oct 30 21:30:08.746895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:08.746866    4918 scope.go:122] "RemoveContainer" containerID="0cb2fd82357ec9ec421baf309ddb3a928fc5fbdc0cc8f2fea183c03880a264f6"
Oct 30 21:30:08.747210 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:08.747186    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz_odh-ai-gateway-infra(778db27c-4f22-4317-9753-b84985424b0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz" podUID="778db27c-4f22-4317-9753-b84985424b0b"
Oct 30 21:30:08.749012 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:08.748994    4918 scope.go:122] "RemoveContainer" containerID="c91db332f1ab74267875d56766f070679bee757cea3bb37ec5892f7c71a22c06"
Oct 30 21:30:08.749272 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:08.749251    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx_odh-ai-gateway-infra(43a51867-ecfe-43e9-b630-b621e9b4e428)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428"
Oct 30 21:30:08.750954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:08.750938    4918 scope.go:122] "RemoveContainer" containerID="95657a11862a7e5069f5846cf96c8aaeffe957270f7a0e44ccb7726a7997b85d"
Oct 30 21:30:08.751164 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:08.751143    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq_odh-ai-gateway-infra(8ea1b87b-1562-4d1e-af32-8078f1e62468)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468"
Oct 30 21:30:09.085572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:09.085494    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac" path="/var/lib/kubelet/pods/e66a5aff-0ac0-4d0c-a83e-74f64adbe3ac/volumes"
Oct 30 21:30:11.038514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.038434    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"]
Oct 30 21:30:11.051577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.051534    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37c7d39-z5hwj"]
Oct 30 21:30:11.083770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.083739    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf394595-f240-4c27-bdf6-ea1f94e74617" path="/var/lib/kubelet/pods/cf394595-f240-4c27-bdf6-ea1f94e74617/volumes"
Oct 30 21:30:11.786499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.786470    4918 generic.go:356] "Generic (PLEG): container finished" podID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerID="cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796" exitCode=0
Oct 30 21:30:11.786750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.786551    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" event={"ID":"1eb115eb-85bb-411f-b199-707dc9df13ba","Type":"ContainerDied","Data":"cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796"}
Oct 30 21:30:11.787092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.787067    4918 scope.go:122] "RemoveContainer" containerID="cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796"
Oct 30 21:30:11.788491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.788470    4918 generic.go:356] "Generic (PLEG): container finished" podID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerID="4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80" exitCode=0
Oct 30 21:30:11.788594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.788543    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" event={"ID":"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2","Type":"ContainerDied","Data":"4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80"}
Oct 30 21:30:11.788949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.788933    4918 scope.go:122] "RemoveContainer" containerID="4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80"
Oct 30 21:30:11.789098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.789071    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:30:11.789179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.789159    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:30:11.789243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.789220    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:30:11.790900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.790867    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:30:11.791004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.790944    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:30:11.791064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:11.791009    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:30:12.171525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:12.171492    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:30:12.854082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:12.854043    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" event={"ID":"1eb115eb-85bb-411f-b199-707dc9df13ba","Type":"ContainerStarted","Data":"85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450"}
Oct 30 21:30:12.857206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:12.857182    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" event={"ID":"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2","Type":"ContainerStarted","Data":"d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e"}
Oct 30 21:30:12.947367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:12.947338    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:30:13.812417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:13.807838    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:30:13.812417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:13.807966    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:30:13.812417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:13.808011    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 21:30:13.941415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:13.937155    4918 patch_prober.go:29] interesting pod/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 21:30:13.941415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:13.937225    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 21:30:14.807417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:14.806975    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:30:14.807417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:14.807043    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 21:30:14.948418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:14.940662    4918 patch_prober.go:29] interesting pod/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 21:30:14.948418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:14.940724    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 21:30:15.809417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:15.808890    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:30:15.809417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:15.808953    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 21:30:15.939832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:15.931786    4918 patch_prober.go:29] interesting pod/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 21:30:15.939832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:15.931847    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 21:30:16.840419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:16.839328    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:30:16.840419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:16.839410    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 21:30:16.946052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:16.945484    4918 patch_prober.go:29] interesting pod/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 21:30:16.946052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:16.945558    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 21:30:17.810416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:17.807788    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:30:17.810416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:17.807868    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 21:30:17.939963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:17.939929    4918 patch_prober.go:29] interesting pod/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 21:30:17.940574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:17.940542    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 21:30:18.077366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:18.077283    4918 scope.go:122] "RemoveContainer" containerID="0cb2fd82357ec9ec421baf309ddb3a928fc5fbdc0cc8f2fea183c03880a264f6"
Oct 30 21:30:18.811476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:18.808842    4918 patch_prober.go:29] interesting pod/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 30 21:30:18.811476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:18.808918    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 30 21:30:18.933852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:18.933815    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:30:18.934574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:18.934547    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:30:18.937436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:18.937357    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:30:19.069360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:19.066646    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz" event={"ID":"778db27c-4f22-4317-9753-b84985424b0b","Type":"ContainerStarted","Data":"6be5c2a9b16d3d3f252972ea1576bbc74343f2cbe4984a7a252230e557bd3aab"}
Oct 30 21:30:19.071062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:19.071034    4918 scope.go:122] "RemoveContainer" containerID="2b496008ba54bbaa2990a30bbb1d5cd2ddbaa423451188874971cd7a2517780b"
Oct 30 21:30:19.071210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:19.071182    4918 scope.go:122] "RemoveContainer" containerID="88f71f65f7ee15edbd802de20ab7b95278bb2e3d829afcffe1736f8f12d65d24"
Oct 30 21:30:19.071984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:19.071959    4918 scope.go:122] "RemoveContainer" containerID="95657a11862a7e5069f5846cf96c8aaeffe957270f7a0e44ccb7726a7997b85d"
Oct 30 21:30:19.810646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:19.810619    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:30:19.811135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:19.811113    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:30:19.811944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:19.811921    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:30:20.070821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:20.070745    4918 scope.go:122] "RemoveContainer" containerID="542dd579d4ac3bedc5cb32230646b2979b6013577896bb77de180b22248a8ca9"
Oct 30 21:30:20.071245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:20.070953    4918 scope.go:122] "RemoveContainer" containerID="e2a3b95d733594f9792569173c834a8a1b9ad8f283eb9ada52d3ef22ddbb60e3"
Oct 30 21:30:20.071245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:20.071093    4918 scope.go:122] "RemoveContainer" containerID="4ad71ebddab5db71ef28040b2d1f51f6e4fb0a5ded493788067f594fad4fa246"
Oct 30 21:30:20.085044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:20.085017    4918 generic.go:356] "Generic (PLEG): container finished" podID="778db27c-4f22-4317-9753-b84985424b0b" containerID="6be5c2a9b16d3d3f252972ea1576bbc74343f2cbe4984a7a252230e557bd3aab" exitCode=60
Oct 30 21:30:20.085175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:20.085086    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz" event={"ID":"778db27c-4f22-4317-9753-b84985424b0b","Type":"ContainerDied","Data":"6be5c2a9b16d3d3f252972ea1576bbc74343f2cbe4984a7a252230e557bd3aab"}
Oct 30 21:30:20.085175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:20.085138    4918 scope.go:122] "RemoveContainer" containerID="0cb2fd82357ec9ec421baf309ddb3a928fc5fbdc0cc8f2fea183c03880a264f6"
Oct 30 21:30:20.085600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:20.085577    4918 scope.go:122] "RemoveContainer" containerID="6be5c2a9b16d3d3f252972ea1576bbc74343f2cbe4984a7a252230e557bd3aab"
Oct 30 21:30:20.085988 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:20.085950    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz_odh-ai-gateway-infra(778db27c-4f22-4317-9753-b84985424b0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz" podUID="778db27c-4f22-4317-9753-b84985424b0b"
Oct 30 21:30:20.088941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:20.088917    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq" event={"ID":"8ea1b87b-1562-4d1e-af32-8078f1e62468","Type":"ContainerStarted","Data":"d1266a8d17a9e9befd0990604f9780477e233eb0fc41b50502837df4cf434e1c"}
Oct 30 21:30:20.094496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:20.094056    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v" event={"ID":"fc27f4e0-65dd-4971-abee-816ac50c9869","Type":"ContainerStarted","Data":"fa4b6dee06a78b49a0c367cfc32e3afe95071b5f395162132c0d27b157e3f19e"}
Oct 30 21:30:20.097341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:20.097319    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv" event={"ID":"c68adc98-00aa-48da-8690-4ce89ca58a57","Type":"ContainerStarted","Data":"57839da3412be0b53777f4df3dd3032e38754144427cb99936b587e91d056b3e"}
Oct 30 21:30:20.101203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:20.101182    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"]
Oct 30 21:30:21.025360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.025314    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"]
Oct 30 21:30:21.030668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.030640    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e10b7-cwq86"]
Oct 30 21:30:21.071103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.071073    4918 scope.go:122] "RemoveContainer" containerID="9be5356266ff4e8c77c851d719be544e5832e55d69786117255fa3ca0ab8cfd3"
Oct 30 21:30:21.071638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.071247    4918 scope.go:122] "RemoveContainer" containerID="61c395261e0962493d0bab12694af5d243e44cf0172a9f1de42e97f56d10691a"
Oct 30 21:30:21.071638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.071380    4918 scope.go:122] "RemoveContainer" containerID="3dc69d1e1ba090da09e5f4150c64d906ebb93d64f138bd6ec2915a6c38d58ac9"
Oct 30 21:30:21.071638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.071482    4918 scope.go:122] "RemoveContainer" containerID="c91db332f1ab74267875d56766f070679bee757cea3bb37ec5892f7c71a22c06"
Oct 30 21:30:21.071876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.071844    4918 scope.go:122] "RemoveContainer" containerID="0dfaea4a4507154b74bc1b6c7bbdd21bc6617ff3ce56bc3d606010ddff58f61c"
Oct 30 21:30:21.082870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.082850    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36bc838-0a59-4ed5-af45-0db8bb269691" path="/var/lib/kubelet/pods/f36bc838-0a59-4ed5-af45-0db8bb269691/volumes"
Oct 30 21:30:21.115743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.115717    4918 generic.go:356] "Generic (PLEG): container finished" podID="8ea1b87b-1562-4d1e-af32-8078f1e62468" containerID="d1266a8d17a9e9befd0990604f9780477e233eb0fc41b50502837df4cf434e1c" exitCode=60
Oct 30 21:30:21.115863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.115786    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq" event={"ID":"8ea1b87b-1562-4d1e-af32-8078f1e62468","Type":"ContainerDied","Data":"d1266a8d17a9e9befd0990604f9780477e233eb0fc41b50502837df4cf434e1c"}
Oct 30 21:30:21.115863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.115839    4918 scope.go:122] "RemoveContainer" containerID="95657a11862a7e5069f5846cf96c8aaeffe957270f7a0e44ccb7726a7997b85d"
Oct 30 21:30:21.116318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.116303    4918 scope.go:122] "RemoveContainer" containerID="d1266a8d17a9e9befd0990604f9780477e233eb0fc41b50502837df4cf434e1c"
Oct 30 21:30:21.116650 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:21.116631    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq_odh-ai-gateway-infra(8ea1b87b-1562-4d1e-af32-8078f1e62468)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468"
Oct 30 21:30:21.118028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.118006    4918 generic.go:356] "Generic (PLEG): container finished" podID="fc27f4e0-65dd-4971-abee-816ac50c9869" containerID="fa4b6dee06a78b49a0c367cfc32e3afe95071b5f395162132c0d27b157e3f19e" exitCode=60
Oct 30 21:30:21.118136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.118085    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v" event={"ID":"fc27f4e0-65dd-4971-abee-816ac50c9869","Type":"ContainerDied","Data":"fa4b6dee06a78b49a0c367cfc32e3afe95071b5f395162132c0d27b157e3f19e"}
Oct 30 21:30:21.118556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.118535    4918 scope.go:122] "RemoveContainer" containerID="fa4b6dee06a78b49a0c367cfc32e3afe95071b5f395162132c0d27b157e3f19e"
Oct 30 21:30:21.118922 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:21.118899    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v_odh-ai-gateway-infra(fc27f4e0-65dd-4971-abee-816ac50c9869)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869"
Oct 30 21:30:21.120201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.120177    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct" event={"ID":"9fbca41f-798b-4b86-b103-47868642c624","Type":"ContainerStarted","Data":"caa2523ba77a54adc5d5cf40f63f1a17c7569be425dc538dbee60457d9195d59"}
Oct 30 21:30:21.123709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.123685    4918 generic.go:356] "Generic (PLEG): container finished" podID="c68adc98-00aa-48da-8690-4ce89ca58a57" containerID="57839da3412be0b53777f4df3dd3032e38754144427cb99936b587e91d056b3e" exitCode=60
Oct 30 21:30:21.123843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.123827    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv" event={"ID":"c68adc98-00aa-48da-8690-4ce89ca58a57","Type":"ContainerDied","Data":"57839da3412be0b53777f4df3dd3032e38754144427cb99936b587e91d056b3e"}
Oct 30 21:30:21.124279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.124258    4918 scope.go:122] "RemoveContainer" containerID="57839da3412be0b53777f4df3dd3032e38754144427cb99936b587e91d056b3e"
Oct 30 21:30:21.124715 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:21.124667    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv_odh-ai-gateway-infra(c68adc98-00aa-48da-8690-4ce89ca58a57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57"
Oct 30 21:30:21.128993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.128966    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh" event={"ID":"17420e84-c277-42b9-99d8-c139e86395df","Type":"ContainerStarted","Data":"e4e3676223a6bd63c46350e472d5aabf40ff5f657e604e7442a5fb0e2fc4b8c3"}
Oct 30 21:30:21.131193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.131171    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65" event={"ID":"1a6c38be-7c77-435c-a127-3d3fd054574e","Type":"ContainerStarted","Data":"58eb7c78e709639cd2cc26c8f5a7852f50cb67e13415bdd03a72b3c57579e412"}
Oct 30 21:30:21.167555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.164379    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"]
Oct 30 21:30:21.183981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.183819    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"]
Oct 30 21:30:21.205520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.205468    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh" podStartSLOduration=21.2054527 podStartE2EDuration="21.2054527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:21.197120163 +0000 UTC m=+1734.243679334" watchObservedRunningTime="2026-10-30 21:30:21.2054527 +0000 UTC m=+1734.252011844"
Oct 30 21:30:21.213612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.213587    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"]
Oct 30 21:30:21.304609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.304586    4918 scope.go:122] "RemoveContainer" containerID="88f71f65f7ee15edbd802de20ab7b95278bb2e3d829afcffe1736f8f12d65d24"
Oct 30 21:30:21.344870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.344843    4918 scope.go:122] "RemoveContainer" containerID="2b496008ba54bbaa2990a30bbb1d5cd2ddbaa423451188874971cd7a2517780b"
Oct 30 21:30:21.357373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.357353    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"
Oct 30 21:30:21.403888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.403859    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/778db27c-4f22-4317-9753-b84985424b0b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/778db27c-4f22-4317-9753-b84985424b0b-maas-api-service-ca\") pod \"778db27c-4f22-4317-9753-b84985424b0b\" (UID: \"778db27c-4f22-4317-9753-b84985424b0b\") "
Oct 30 21:30:21.404077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.403921    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778db27c-4f22-4317-9753-b84985424b0b-kube-api-access-7q8x8\" (UniqueName: \"kubernetes.io/projected/778db27c-4f22-4317-9753-b84985424b0b-kube-api-access-7q8x8\") pod \"778db27c-4f22-4317-9753-b84985424b0b\" (UID: \"778db27c-4f22-4317-9753-b84985424b0b\") "
Oct 30 21:30:21.404318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.404289    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778db27c-4f22-4317-9753-b84985424b0b-maas-api-service-ca" pod "778db27c-4f22-4317-9753-b84985424b0b" (UID: "778db27c-4f22-4317-9753-b84985424b0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:21.404736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.404711    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/778db27c-4f22-4317-9753-b84985424b0b-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:21.525891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.525788    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778db27c-4f22-4317-9753-b84985424b0b-kube-api-access-7q8x8" pod "778db27c-4f22-4317-9753-b84985424b0b" (UID: "778db27c-4f22-4317-9753-b84985424b0b"). InnerVolumeSpecName "kube-api-access-7q8x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:21.627019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:21.626986    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q8x8\" (UniqueName: \"kubernetes.io/projected/778db27c-4f22-4317-9753-b84985424b0b-kube-api-access-7q8x8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.148638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.148607    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg" event={"ID":"8f43a597-0a8a-4c75-8123-d5450be93d21","Type":"ContainerStarted","Data":"98b173f7aa768c52791534ea124960ed63d12b18b5b06a50c0863ff5deafe3b7"}
Oct 30 21:30:22.150725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.150692    4918 generic.go:356] "Generic (PLEG): container finished" podID="17420e84-c277-42b9-99d8-c139e86395df" containerID="e4e3676223a6bd63c46350e472d5aabf40ff5f657e604e7442a5fb0e2fc4b8c3" exitCode=60
Oct 30 21:30:22.150845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.150765    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh" event={"ID":"17420e84-c277-42b9-99d8-c139e86395df","Type":"ContainerDied","Data":"e4e3676223a6bd63c46350e472d5aabf40ff5f657e604e7442a5fb0e2fc4b8c3"}
Oct 30 21:30:22.150845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.150816    4918 scope.go:122] "RemoveContainer" containerID="e2a3b95d733594f9792569173c834a8a1b9ad8f283eb9ada52d3ef22ddbb60e3"
Oct 30 21:30:22.151194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.151167    4918 scope.go:122] "RemoveContainer" containerID="e4e3676223a6bd63c46350e472d5aabf40ff5f657e604e7442a5fb0e2fc4b8c3"
Oct 30 21:30:22.151620 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:22.151592    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh_odh-ai-gateway-infra(17420e84-c277-42b9-99d8-c139e86395df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh" podUID="17420e84-c277-42b9-99d8-c139e86395df"
Oct 30 21:30:22.153140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.153120    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg" event={"ID":"379363eb-5d66-41cf-989c-08cf4a2008ce","Type":"ContainerStarted","Data":"0ac1006643811c41a9dc2a9842f30fff54c990636929207a71d71d7733477591"}
Oct 30 21:30:22.155300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.155279    4918 generic.go:356] "Generic (PLEG): container finished" podID="1a6c38be-7c77-435c-a127-3d3fd054574e" containerID="58eb7c78e709639cd2cc26c8f5a7852f50cb67e13415bdd03a72b3c57579e412" exitCode=60
Oct 30 21:30:22.155500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.155469    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65" event={"ID":"1a6c38be-7c77-435c-a127-3d3fd054574e","Type":"ContainerDied","Data":"58eb7c78e709639cd2cc26c8f5a7852f50cb67e13415bdd03a72b3c57579e412"}
Oct 30 21:30:22.155896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.155868    4918 scope.go:122] "RemoveContainer" containerID="58eb7c78e709639cd2cc26c8f5a7852f50cb67e13415bdd03a72b3c57579e412"
Oct 30 21:30:22.156248 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:22.156219    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65_odh-ai-gateway-infra(1a6c38be-7c77-435c-a127-3d3fd054574e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e"
Oct 30 21:30:22.157026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.156996    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz" event={"ID":"778db27c-4f22-4317-9753-b84985424b0b","Type":"ContainerDied","Data":"41fafc8a4d0421c6418c75ddbd18fbc18f6cb71c638019ef94e9d0444edf9fd3"}
Oct 30 21:30:22.157121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.157067    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"
Oct 30 21:30:22.166637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.166613    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"]
Oct 30 21:30:22.177746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.177715    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx" event={"ID":"43a51867-ecfe-43e9-b630-b621e9b4e428","Type":"ContainerStarted","Data":"51d10139eda1b0c14ceea25ce8b226b55a6642139a34af0dbf6683027e9c98b6"}
Oct 30 21:30:22.181483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.181458    4918 scope.go:122] "RemoveContainer" containerID="542dd579d4ac3bedc5cb32230646b2979b6013577896bb77de180b22248a8ca9"
Oct 30 21:30:22.188781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.188753    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29" event={"ID":"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418","Type":"ContainerStarted","Data":"a8c958a5fa7b9aed1cdb11f0d54d22f2213f3c6e5c572c86793dc65f62f9c779"}
Oct 30 21:30:22.196066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.196040    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt" event={"ID":"e1225dbb-da72-4c3f-b13d-b63d71e3ec51","Type":"ContainerStarted","Data":"527758fdb9e491eb2065b39fa20dfd9327f4f52cce258fdfd4fdf4c697e0ba37"}
Oct 30 21:30:22.203171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.203144    4918 generic.go:356] "Generic (PLEG): container finished" podID="9fbca41f-798b-4b86-b103-47868642c624" containerID="caa2523ba77a54adc5d5cf40f63f1a17c7569be425dc538dbee60457d9195d59" exitCode=60
Oct 30 21:30:22.203291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.203202    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct" event={"ID":"9fbca41f-798b-4b86-b103-47868642c624","Type":"ContainerDied","Data":"caa2523ba77a54adc5d5cf40f63f1a17c7569be425dc538dbee60457d9195d59"}
Oct 30 21:30:22.211320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.211302    4918 scope.go:122] "RemoveContainer" containerID="6be5c2a9b16d3d3f252972ea1576bbc74343f2cbe4984a7a252230e557bd3aab"
Oct 30 21:30:22.218281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.218260    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"]
Oct 30 21:30:22.258453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.248382    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"]
Oct 30 21:30:22.266310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.266024    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg" podStartSLOduration=22.266006558 podStartE2EDuration="22.266006558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:22.257205919 +0000 UTC m=+1735.303765095" watchObservedRunningTime="2026-10-30 21:30:22.266006558 +0000 UTC m=+1735.312565704"
Oct 30 21:30:22.269117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.268070    4918 scope.go:122] "RemoveContainer" containerID="4ad71ebddab5db71ef28040b2d1f51f6e4fb0a5ded493788067f594fad4fa246"
Oct 30 21:30:22.360908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.360860    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"]
Oct 30 21:30:22.409779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.401331    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a33b12-29889930-vldmz"]
Oct 30 21:30:22.778108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.778078    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"
Oct 30 21:30:22.781963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.781939    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"
Oct 30 21:30:22.786350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.786327    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"
Oct 30 21:30:22.790444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.790428    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"
Oct 30 21:30:22.875367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.875332    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbca41f-798b-4b86-b103-47868642c624-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbca41f-798b-4b86-b103-47868642c624-maas-api-service-ca\") pod \"9fbca41f-798b-4b86-b103-47868642c624\" (UID: \"9fbca41f-798b-4b86-b103-47868642c624\") "
Oct 30 21:30:22.875577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.875506    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbca41f-798b-4b86-b103-47868642c624-kube-api-access-qfpfh\" (UniqueName: \"kubernetes.io/projected/9fbca41f-798b-4b86-b103-47868642c624-kube-api-access-qfpfh\") pod \"9fbca41f-798b-4b86-b103-47868642c624\" (UID: \"9fbca41f-798b-4b86-b103-47868642c624\") "
Oct 30 21:30:22.875671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.875606    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea1b87b-1562-4d1e-af32-8078f1e62468-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea1b87b-1562-4d1e-af32-8078f1e62468-maas-api-service-ca\") pod \"8ea1b87b-1562-4d1e-af32-8078f1e62468\" (UID: \"8ea1b87b-1562-4d1e-af32-8078f1e62468\") "
Oct 30 21:30:22.875671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.875630    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c68adc98-00aa-48da-8690-4ce89ca58a57-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68adc98-00aa-48da-8690-4ce89ca58a57-maas-api-service-ca\") pod \"c68adc98-00aa-48da-8690-4ce89ca58a57\" (UID: \"c68adc98-00aa-48da-8690-4ce89ca58a57\") "
Oct 30 21:30:22.875785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.875714    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc27f4e0-65dd-4971-abee-816ac50c9869-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc27f4e0-65dd-4971-abee-816ac50c9869-maas-api-service-ca\") pod \"fc27f4e0-65dd-4971-abee-816ac50c9869\" (UID: \"fc27f4e0-65dd-4971-abee-816ac50c9869\") "
Oct 30 21:30:22.875785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.875746    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68adc98-00aa-48da-8690-4ce89ca58a57-kube-api-access-7b657\" (UniqueName: \"kubernetes.io/projected/c68adc98-00aa-48da-8690-4ce89ca58a57-kube-api-access-7b657\") pod \"c68adc98-00aa-48da-8690-4ce89ca58a57\" (UID: \"c68adc98-00aa-48da-8690-4ce89ca58a57\") "
Oct 30 21:30:22.875785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.875761    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbca41f-798b-4b86-b103-47868642c624-maas-api-service-ca" pod "9fbca41f-798b-4b86-b103-47868642c624" (UID: "9fbca41f-798b-4b86-b103-47868642c624"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:22.875950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.875808    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea1b87b-1562-4d1e-af32-8078f1e62468-kube-api-access-lxdjb\" (UniqueName: \"kubernetes.io/projected/8ea1b87b-1562-4d1e-af32-8078f1e62468-kube-api-access-lxdjb\") pod \"8ea1b87b-1562-4d1e-af32-8078f1e62468\" (UID: \"8ea1b87b-1562-4d1e-af32-8078f1e62468\") "
Oct 30 21:30:22.875950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.875835    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc27f4e0-65dd-4971-abee-816ac50c9869-kube-api-access-7n7rn\" (UniqueName: \"kubernetes.io/projected/fc27f4e0-65dd-4971-abee-816ac50c9869-kube-api-access-7n7rn\") pod \"fc27f4e0-65dd-4971-abee-816ac50c9869\" (UID: \"fc27f4e0-65dd-4971-abee-816ac50c9869\") "
Oct 30 21:30:22.876164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.876138    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68adc98-00aa-48da-8690-4ce89ca58a57-maas-api-service-ca" pod "c68adc98-00aa-48da-8690-4ce89ca58a57" (UID: "c68adc98-00aa-48da-8690-4ce89ca58a57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:22.876540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.876513    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc27f4e0-65dd-4971-abee-816ac50c9869-maas-api-service-ca" pod "fc27f4e0-65dd-4971-abee-816ac50c9869" (UID: "fc27f4e0-65dd-4971-abee-816ac50c9869"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:22.876637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.876543    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea1b87b-1562-4d1e-af32-8078f1e62468-maas-api-service-ca" pod "8ea1b87b-1562-4d1e-af32-8078f1e62468" (UID: "8ea1b87b-1562-4d1e-af32-8078f1e62468"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:22.876911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.876885    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ea1b87b-1562-4d1e-af32-8078f1e62468-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.877005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.876911    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c68adc98-00aa-48da-8690-4ce89ca58a57-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.877005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.876926    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc27f4e0-65dd-4971-abee-816ac50c9869-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.877005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.876942    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbca41f-798b-4b86-b103-47868642c624-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.878119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.878098    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbca41f-798b-4b86-b103-47868642c624-kube-api-access-qfpfh" pod "9fbca41f-798b-4b86-b103-47868642c624" (UID: "9fbca41f-798b-4b86-b103-47868642c624"). InnerVolumeSpecName "kube-api-access-qfpfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:22.878729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.878708    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68adc98-00aa-48da-8690-4ce89ca58a57-kube-api-access-7b657" pod "c68adc98-00aa-48da-8690-4ce89ca58a57" (UID: "c68adc98-00aa-48da-8690-4ce89ca58a57"). InnerVolumeSpecName "kube-api-access-7b657". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:22.878983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.878959    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea1b87b-1562-4d1e-af32-8078f1e62468-kube-api-access-lxdjb" pod "8ea1b87b-1562-4d1e-af32-8078f1e62468" (UID: "8ea1b87b-1562-4d1e-af32-8078f1e62468"). InnerVolumeSpecName "kube-api-access-lxdjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:22.879059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.879035    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc27f4e0-65dd-4971-abee-816ac50c9869-kube-api-access-7n7rn" pod "fc27f4e0-65dd-4971-abee-816ac50c9869" (UID: "fc27f4e0-65dd-4971-abee-816ac50c9869"). InnerVolumeSpecName "kube-api-access-7n7rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:22.928823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.928744    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"]
Oct 30 21:30:22.978511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.978476    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b657\" (UniqueName: \"kubernetes.io/projected/c68adc98-00aa-48da-8690-4ce89ca58a57-kube-api-access-7b657\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.978511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.978502    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxdjb\" (UniqueName: \"kubernetes.io/projected/8ea1b87b-1562-4d1e-af32-8078f1e62468-kube-api-access-lxdjb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.978511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.978512    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n7rn\" (UniqueName: \"kubernetes.io/projected/fc27f4e0-65dd-4971-abee-816ac50c9869-kube-api-access-7n7rn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:22.978823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:22.978524    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfpfh\" (UniqueName: \"kubernetes.io/projected/9fbca41f-798b-4b86-b103-47868642c624-kube-api-access-qfpfh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:23.097585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.097544    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="778db27c-4f22-4317-9753-b84985424b0b" path="/var/lib/kubelet/pods/778db27c-4f22-4317-9753-b84985424b0b/volumes"
Oct 30 21:30:23.219160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.219129    4918 generic.go:356] "Generic (PLEG): container finished" podID="43a51867-ecfe-43e9-b630-b621e9b4e428" containerID="51d10139eda1b0c14ceea25ce8b226b55a6642139a34af0dbf6683027e9c98b6" exitCode=60
Oct 30 21:30:23.219875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.219237    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx" event={"ID":"43a51867-ecfe-43e9-b630-b621e9b4e428","Type":"ContainerDied","Data":"51d10139eda1b0c14ceea25ce8b226b55a6642139a34af0dbf6683027e9c98b6"}
Oct 30 21:30:23.219875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.219294    4918 scope.go:122] "RemoveContainer" containerID="c91db332f1ab74267875d56766f070679bee757cea3bb37ec5892f7c71a22c06"
Oct 30 21:30:23.219875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.219821    4918 scope.go:122] "RemoveContainer" containerID="51d10139eda1b0c14ceea25ce8b226b55a6642139a34af0dbf6683027e9c98b6"
Oct 30 21:30:23.220258 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:23.220231    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx_odh-ai-gateway-infra(43a51867-ecfe-43e9-b630-b621e9b4e428)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428"
Oct 30 21:30:23.222629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.222298    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq" event={"ID":"8ea1b87b-1562-4d1e-af32-8078f1e62468","Type":"ContainerDied","Data":"ebe27b83a17fbef729e27d403ecae7f062e46d66902e253a724f6e51ee142e3c"}
Oct 30 21:30:23.222629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.222462    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"
Oct 30 21:30:23.332794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.328703    4918 generic.go:356] "Generic (PLEG): container finished" podID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" containerID="a8c958a5fa7b9aed1cdb11f0d54d22f2213f3c6e5c572c86793dc65f62f9c779" exitCode=60
Oct 30 21:30:23.332794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.328872    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29" event={"ID":"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418","Type":"ContainerDied","Data":"a8c958a5fa7b9aed1cdb11f0d54d22f2213f3c6e5c572c86793dc65f62f9c779"}
Oct 30 21:30:23.342424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.338012    4918 scope.go:122] "RemoveContainer" containerID="a8c958a5fa7b9aed1cdb11f0d54d22f2213f3c6e5c572c86793dc65f62f9c779"
Oct 30 21:30:23.342424 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:23.338547    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-kwv29_odh-ai-gateway-infra(0a9f2c2a-8aea-4ecf-941b-de85d3bc6418)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418"
Oct 30 21:30:23.412417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.408718    4918 generic.go:356] "Generic (PLEG): container finished" podID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" containerID="527758fdb9e491eb2065b39fa20dfd9327f4f52cce258fdfd4fdf4c697e0ba37" exitCode=60
Oct 30 21:30:23.412417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.408866    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt" event={"ID":"e1225dbb-da72-4c3f-b13d-b63d71e3ec51","Type":"ContainerDied","Data":"527758fdb9e491eb2065b39fa20dfd9327f4f52cce258fdfd4fdf4c697e0ba37"}
Oct 30 21:30:23.412417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.409270    4918 scope.go:122] "RemoveContainer" containerID="527758fdb9e491eb2065b39fa20dfd9327f4f52cce258fdfd4fdf4c697e0ba37"
Oct 30 21:30:23.412417 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:23.409736    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt_odh-ai-gateway-infra(e1225dbb-da72-4c3f-b13d-b63d71e3ec51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51"
Oct 30 21:30:23.460412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.457187    4918 scope.go:122] "RemoveContainer" containerID="d1266a8d17a9e9befd0990604f9780477e233eb0fc41b50502837df4cf434e1c"
Oct 30 21:30:23.521415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.515819    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v" event={"ID":"fc27f4e0-65dd-4971-abee-816ac50c9869","Type":"ContainerDied","Data":"c8be3d8116063825c18e48a86e31bb8338afba51320625c3b3cc6d0828736ff7"}
Oct 30 21:30:23.521415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.515984    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"
Oct 30 21:30:23.564416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.563558    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct" event={"ID":"9fbca41f-798b-4b86-b103-47868642c624","Type":"ContainerDied","Data":"edc8f169412983e31fe7ed85208960c918bbc9ceca363eb7cb834c1b05e5a45f"}
Oct 30 21:30:23.564416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.563695    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"
Oct 30 21:30:23.594961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.594930    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"
Oct 30 21:30:23.617079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.597518    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"]
Oct 30 21:30:23.617079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.597589    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv" event={"ID":"c68adc98-00aa-48da-8690-4ce89ca58a57","Type":"ContainerDied","Data":"04e4b7877ba227831c747d0361ac774ae33b6bd69be23d415814b49b6b318f01"}
Oct 30 21:30:23.637038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.637002    4918 scope.go:122] "RemoveContainer" containerID="61c395261e0962493d0bab12694af5d243e44cf0172a9f1de42e97f56d10691a"
Oct 30 21:30:23.639148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.639120    4918 generic.go:356] "Generic (PLEG): container finished" podID="8f43a597-0a8a-4c75-8123-d5450be93d21" containerID="98b173f7aa768c52791534ea124960ed63d12b18b5b06a50c0863ff5deafe3b7" exitCode=60
Oct 30 21:30:23.639429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.639367    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg" event={"ID":"8f43a597-0a8a-4c75-8123-d5450be93d21","Type":"ContainerDied","Data":"98b173f7aa768c52791534ea124960ed63d12b18b5b06a50c0863ff5deafe3b7"}
Oct 30 21:30:23.665098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.665064    4918 generic.go:356] "Generic (PLEG): container finished" podID="379363eb-5d66-41cf-989c-08cf4a2008ce" containerID="0ac1006643811c41a9dc2a9842f30fff54c990636929207a71d71d7733477591" exitCode=60
Oct 30 21:30:23.665429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.665407    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg" event={"ID":"379363eb-5d66-41cf-989c-08cf4a2008ce","Type":"ContainerDied","Data":"0ac1006643811c41a9dc2a9842f30fff54c990636929207a71d71d7733477591"}
Oct 30 21:30:23.682550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.666173    4918 scope.go:122] "RemoveContainer" containerID="0ac1006643811c41a9dc2a9842f30fff54c990636929207a71d71d7733477591"
Oct 30 21:30:23.715995 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:23.701934    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg_odh-ai-gateway-infra(379363eb-5d66-41cf-989c-08cf4a2008ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce"
Oct 30 21:30:23.716447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.713348    4918 scope.go:122] "RemoveContainer" containerID="0dfaea4a4507154b74bc1b6c7bbdd21bc6617ff3ce56bc3d606010ddff58f61c"
Oct 30 21:30:23.722727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.722699    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"]
Oct 30 21:30:23.749283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.749252    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"]
Oct 30 21:30:23.780307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.780147    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"]
Oct 30 21:30:23.782944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.782835    4918 scope.go:122] "RemoveContainer" containerID="fa4b6dee06a78b49a0c367cfc32e3afe95071b5f395162132c0d27b157e3f19e"
Oct 30 21:30:23.799815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.799779    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"]
Oct 30 21:30:23.830689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.830624    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"]
Oct 30 21:30:23.836152 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.834737    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-46c98c-29889930-nftbq"]
Oct 30 21:30:23.850593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.850549    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"]
Oct 30 21:30:23.854525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.854362    4918 scope.go:122] "RemoveContainer" containerID="caa2523ba77a54adc5d5cf40f63f1a17c7569be425dc538dbee60457d9195d59"
Oct 30 21:30:23.872548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.872477    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-128d2f-29889930-thbqv"]
Oct 30 21:30:23.891446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.884153    4918 scope.go:122] "RemoveContainer" containerID="57839da3412be0b53777f4df3dd3032e38754144427cb99936b587e91d056b3e"
Oct 30 21:30:23.898015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.897961    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"]
Oct 30 21:30:23.909207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.909152    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889930-nxvct"]
Oct 30 21:30:23.924650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.924558    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"]
Oct 30 21:30:23.930200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.929652    4918 scope.go:122] "RemoveContainer" containerID="3dc69d1e1ba090da09e5f4150c64d906ebb93d64f138bd6ec2915a6c38d58ac9"
Oct 30 21:30:23.939833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.938870    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d4a0a5-29889930-m5m9v"]
Oct 30 21:30:23.990892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.990867    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"
Oct 30 21:30:23.994459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:23.994372    4918 scope.go:122] "RemoveContainer" containerID="9be5356266ff4e8c77c851d719be544e5832e55d69786117255fa3ca0ab8cfd3"
Oct 30 21:30:24.043300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.043198    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a6c38be-7c77-435c-a127-3d3fd054574e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6c38be-7c77-435c-a127-3d3fd054574e-maas-api-service-ca\") pod \"1a6c38be-7c77-435c-a127-3d3fd054574e\" (UID: \"1a6c38be-7c77-435c-a127-3d3fd054574e\") "
Oct 30 21:30:24.043515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.043372    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6c38be-7c77-435c-a127-3d3fd054574e-kube-api-access-z6wvv\" (UniqueName: \"kubernetes.io/projected/1a6c38be-7c77-435c-a127-3d3fd054574e-kube-api-access-z6wvv\") pod \"1a6c38be-7c77-435c-a127-3d3fd054574e\" (UID: \"1a6c38be-7c77-435c-a127-3d3fd054574e\") "
Oct 30 21:30:24.044627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.044600    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6c38be-7c77-435c-a127-3d3fd054574e-maas-api-service-ca" pod "1a6c38be-7c77-435c-a127-3d3fd054574e" (UID: "1a6c38be-7c77-435c-a127-3d3fd054574e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:24.049073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.049049    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6c38be-7c77-435c-a127-3d3fd054574e-kube-api-access-z6wvv" pod "1a6c38be-7c77-435c-a127-3d3fd054574e" (UID: "1a6c38be-7c77-435c-a127-3d3fd054574e"). InnerVolumeSpecName "kube-api-access-z6wvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:24.128068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.128042    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"
Oct 30 21:30:24.130802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.130783    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"
Oct 30 21:30:24.146314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.146291    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6wvv\" (UniqueName: \"kubernetes.io/projected/1a6c38be-7c77-435c-a127-3d3fd054574e-kube-api-access-z6wvv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.146411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.146319    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a6c38be-7c77-435c-a127-3d3fd054574e-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.247653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.247625    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17420e84-c277-42b9-99d8-c139e86395df-kube-api-access-4sfn9\" (UniqueName: \"kubernetes.io/projected/17420e84-c277-42b9-99d8-c139e86395df-kube-api-access-4sfn9\") pod \"17420e84-c277-42b9-99d8-c139e86395df\" (UID: \"17420e84-c277-42b9-99d8-c139e86395df\") "
Oct 30 21:30:24.248172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.247720    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17420e84-c277-42b9-99d8-c139e86395df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17420e84-c277-42b9-99d8-c139e86395df-maas-api-service-ca\") pod \"17420e84-c277-42b9-99d8-c139e86395df\" (UID: \"17420e84-c277-42b9-99d8-c139e86395df\") "
Oct 30 21:30:24.248172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.247846    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f43a597-0a8a-4c75-8123-d5450be93d21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f43a597-0a8a-4c75-8123-d5450be93d21-maas-api-service-ca\") pod \"8f43a597-0a8a-4c75-8123-d5450be93d21\" (UID: \"8f43a597-0a8a-4c75-8123-d5450be93d21\") "
Oct 30 21:30:24.248172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.248003    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f43a597-0a8a-4c75-8123-d5450be93d21-kube-api-access-5gj8g\" (UniqueName: \"kubernetes.io/projected/8f43a597-0a8a-4c75-8123-d5450be93d21-kube-api-access-5gj8g\") pod \"8f43a597-0a8a-4c75-8123-d5450be93d21\" (UID: \"8f43a597-0a8a-4c75-8123-d5450be93d21\") "
Oct 30 21:30:24.248172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.248136    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17420e84-c277-42b9-99d8-c139e86395df-maas-api-service-ca" pod "17420e84-c277-42b9-99d8-c139e86395df" (UID: "17420e84-c277-42b9-99d8-c139e86395df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:24.248348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.248214    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f43a597-0a8a-4c75-8123-d5450be93d21-maas-api-service-ca" pod "8f43a597-0a8a-4c75-8123-d5450be93d21" (UID: "8f43a597-0a8a-4c75-8123-d5450be93d21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:24.248719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.248701    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f43a597-0a8a-4c75-8123-d5450be93d21-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.248773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.248725    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17420e84-c277-42b9-99d8-c139e86395df-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.250059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.250035    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17420e84-c277-42b9-99d8-c139e86395df-kube-api-access-4sfn9" pod "17420e84-c277-42b9-99d8-c139e86395df" (UID: "17420e84-c277-42b9-99d8-c139e86395df"). InnerVolumeSpecName "kube-api-access-4sfn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:24.250187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.250172    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f43a597-0a8a-4c75-8123-d5450be93d21-kube-api-access-5gj8g" pod "8f43a597-0a8a-4c75-8123-d5450be93d21" (UID: "8f43a597-0a8a-4c75-8123-d5450be93d21"). InnerVolumeSpecName "kube-api-access-5gj8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:24.350459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.350350    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sfn9\" (UniqueName: \"kubernetes.io/projected/17420e84-c277-42b9-99d8-c139e86395df-kube-api-access-4sfn9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.350459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.350378    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gj8g\" (UniqueName: \"kubernetes.io/projected/8f43a597-0a8a-4c75-8123-d5450be93d21-kube-api-access-5gj8g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:24.681274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.681246    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"
Oct 30 21:30:24.681274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.681262    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65" event={"ID":"1a6c38be-7c77-435c-a127-3d3fd054574e","Type":"ContainerDied","Data":"6333b4b2fe7ae4ac6588c8cb4924690261b4ceece0e0143214c319ac2e25815c"}
Oct 30 21:30:24.681516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.681362    4918 scope.go:122] "RemoveContainer" containerID="58eb7c78e709639cd2cc26c8f5a7852f50cb67e13415bdd03a72b3c57579e412"
Oct 30 21:30:24.697592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.697504    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"
Oct 30 21:30:24.697592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.697546    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg" event={"ID":"8f43a597-0a8a-4c75-8123-d5450be93d21","Type":"ContainerDied","Data":"8f19bcb17e5b898913a4b57ed67c639428994a0536803c954f6dc640a6117f3d"}
Oct 30 21:30:24.697784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.697610    4918 scope.go:122] "RemoveContainer" containerID="98b173f7aa768c52791534ea124960ed63d12b18b5b06a50c0863ff5deafe3b7"
Oct 30 21:30:24.702949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.702908    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh" event={"ID":"17420e84-c277-42b9-99d8-c139e86395df","Type":"ContainerDied","Data":"7787f75ad68c6bd061d89f4a153f0147b5d47b793ed561188841376c08d6bdc8"}
Oct 30 21:30:24.703086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.703022    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"
Oct 30 21:30:24.717173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.717148    4918 scope.go:122] "RemoveContainer" containerID="e4e3676223a6bd63c46350e472d5aabf40ff5f657e604e7442a5fb0e2fc4b8c3"
Oct 30 21:30:24.723096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.723034    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"]
Oct 30 21:30:24.737411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.730906    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889930-89v65"]
Oct 30 21:30:24.811723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.811693    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"]
Oct 30 21:30:24.821100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.821072    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-6ed261-29889930-xcfxh"]
Oct 30 21:30:24.830658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.830632    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"]
Oct 30 21:30:24.835586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.835561    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-05a342-29889930-qbbjg"]
Oct 30 21:30:24.916501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.916380    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"
Oct 30 21:30:24.992924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.992896    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/379363eb-5d66-41cf-989c-08cf4a2008ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/379363eb-5d66-41cf-989c-08cf4a2008ce-maas-api-service-ca\") pod \"379363eb-5d66-41cf-989c-08cf4a2008ce\" (UID: \"379363eb-5d66-41cf-989c-08cf4a2008ce\") "
Oct 30 21:30:24.993628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.993604    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379363eb-5d66-41cf-989c-08cf4a2008ce-kube-api-access-d46wp\" (UniqueName: \"kubernetes.io/projected/379363eb-5d66-41cf-989c-08cf4a2008ce-kube-api-access-d46wp\") pod \"379363eb-5d66-41cf-989c-08cf4a2008ce\" (UID: \"379363eb-5d66-41cf-989c-08cf4a2008ce\") "
Oct 30 21:30:24.993857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.993780    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379363eb-5d66-41cf-989c-08cf4a2008ce-maas-api-service-ca" pod "379363eb-5d66-41cf-989c-08cf4a2008ce" (UID: "379363eb-5d66-41cf-989c-08cf4a2008ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:24.994642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:24.994626    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/379363eb-5d66-41cf-989c-08cf4a2008ce-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.006240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.006210    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379363eb-5d66-41cf-989c-08cf4a2008ce-kube-api-access-d46wp" pod "379363eb-5d66-41cf-989c-08cf4a2008ce" (UID: "379363eb-5d66-41cf-989c-08cf4a2008ce"). InnerVolumeSpecName "kube-api-access-d46wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:25.042715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.042684    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"
Oct 30 21:30:25.046311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.046281    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"
Oct 30 21:30:25.050537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.050517    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"
Oct 30 21:30:25.091337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.091302    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17420e84-c277-42b9-99d8-c139e86395df" path="/var/lib/kubelet/pods/17420e84-c277-42b9-99d8-c139e86395df/volumes"
Oct 30 21:30:25.092056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.092036    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e" path="/var/lib/kubelet/pods/1a6c38be-7c77-435c-a127-3d3fd054574e/volumes"
Oct 30 21:30:25.092687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.092668    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468" path="/var/lib/kubelet/pods/8ea1b87b-1562-4d1e-af32-8078f1e62468/volumes"
Oct 30 21:30:25.093719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.093696    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f43a597-0a8a-4c75-8123-d5450be93d21" path="/var/lib/kubelet/pods/8f43a597-0a8a-4c75-8123-d5450be93d21/volumes"
Oct 30 21:30:25.094294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.094275    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbca41f-798b-4b86-b103-47868642c624" path="/var/lib/kubelet/pods/9fbca41f-798b-4b86-b103-47868642c624/volumes"
Oct 30 21:30:25.094952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.094932    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57" path="/var/lib/kubelet/pods/c68adc98-00aa-48da-8690-4ce89ca58a57/volumes"
Oct 30 21:30:25.096159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.096139    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869" path="/var/lib/kubelet/pods/fc27f4e0-65dd-4971-abee-816ac50c9869/volumes"
Oct 30 21:30:25.101787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.101762    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d46wp\" (UniqueName: \"kubernetes.io/projected/379363eb-5d66-41cf-989c-08cf4a2008ce-kube-api-access-d46wp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.210635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.207935    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-kube-api-access-l67kc\" (UniqueName: \"kubernetes.io/projected/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-kube-api-access-l67kc\") pod \"e1225dbb-da72-4c3f-b13d-b63d71e3ec51\" (UID: \"e1225dbb-da72-4c3f-b13d-b63d71e3ec51\") "
Oct 30 21:30:25.210635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.208043    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-kube-api-access-7jhfp\" (UniqueName: \"kubernetes.io/projected/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-kube-api-access-7jhfp\") pod \"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418\" (UID: \"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418\") "
Oct 30 21:30:25.210635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.208378    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-maas-api-service-ca\") pod \"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418\" (UID: \"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418\") "
Oct 30 21:30:25.210635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.208489    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a51867-ecfe-43e9-b630-b621e9b4e428-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a51867-ecfe-43e9-b630-b621e9b4e428-maas-api-service-ca\") pod \"43a51867-ecfe-43e9-b630-b621e9b4e428\" (UID: \"43a51867-ecfe-43e9-b630-b621e9b4e428\") "
Oct 30 21:30:25.213901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.213869    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-maas-api-service-ca" pod "0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" (UID: "0a9f2c2a-8aea-4ecf-941b-de85d3bc6418"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:25.214384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.208534    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-maas-api-service-ca\") pod \"e1225dbb-da72-4c3f-b13d-b63d71e3ec51\" (UID: \"e1225dbb-da72-4c3f-b13d-b63d71e3ec51\") "
Oct 30 21:30:25.214603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.214587    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a51867-ecfe-43e9-b630-b621e9b4e428-kube-api-access-ks8cf\" (UniqueName: \"kubernetes.io/projected/43a51867-ecfe-43e9-b630-b621e9b4e428-kube-api-access-ks8cf\") pod \"43a51867-ecfe-43e9-b630-b621e9b4e428\" (UID: \"43a51867-ecfe-43e9-b630-b621e9b4e428\") "
Oct 30 21:30:25.215949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.215893    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.217764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.217739    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a51867-ecfe-43e9-b630-b621e9b4e428-maas-api-service-ca" pod "43a51867-ecfe-43e9-b630-b621e9b4e428" (UID: "43a51867-ecfe-43e9-b630-b621e9b4e428"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:25.218241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.218223    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-maas-api-service-ca" pod "e1225dbb-da72-4c3f-b13d-b63d71e3ec51" (UID: "e1225dbb-da72-4c3f-b13d-b63d71e3ec51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:25.249777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.249688    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-kube-api-access-l67kc" pod "e1225dbb-da72-4c3f-b13d-b63d71e3ec51" (UID: "e1225dbb-da72-4c3f-b13d-b63d71e3ec51"). InnerVolumeSpecName "kube-api-access-l67kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:25.250703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.250664    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a51867-ecfe-43e9-b630-b621e9b4e428-kube-api-access-ks8cf" pod "43a51867-ecfe-43e9-b630-b621e9b4e428" (UID: "43a51867-ecfe-43e9-b630-b621e9b4e428"). InnerVolumeSpecName "kube-api-access-ks8cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:25.255484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.255454    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-kube-api-access-7jhfp" pod "0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" (UID: "0a9f2c2a-8aea-4ecf-941b-de85d3bc6418"). InnerVolumeSpecName "kube-api-access-7jhfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:25.317562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.317533    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a51867-ecfe-43e9-b630-b621e9b4e428-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.317562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.317559    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.317773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.317571    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8cf\" (UniqueName: \"kubernetes.io/projected/43a51867-ecfe-43e9-b630-b621e9b4e428-kube-api-access-ks8cf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.317773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.317580    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l67kc\" (UniqueName: \"kubernetes.io/projected/e1225dbb-da72-4c3f-b13d-b63d71e3ec51-kube-api-access-l67kc\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.317773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.317589    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jhfp\" (UniqueName: \"kubernetes.io/projected/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418-kube-api-access-7jhfp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:25.716466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.716341    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"
Oct 30 21:30:25.716466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.716358    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29" event={"ID":"0a9f2c2a-8aea-4ecf-941b-de85d3bc6418","Type":"ContainerDied","Data":"b4c5549fdd79ad2f91f5e56ee51770c4e09c30e10cf226368de698072295ccc0"}
Oct 30 21:30:25.716712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.716465    4918 scope.go:122] "RemoveContainer" containerID="a8c958a5fa7b9aed1cdb11f0d54d22f2213f3c6e5c572c86793dc65f62f9c779"
Oct 30 21:30:25.718028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.718002    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt" event={"ID":"e1225dbb-da72-4c3f-b13d-b63d71e3ec51","Type":"ContainerDied","Data":"b295adce2deccb70b86e5f95e7837876da7e77745c839fdee30924ad504cf1ec"}
Oct 30 21:30:25.718130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.718020    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"
Oct 30 21:30:25.730978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.730921    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg" event={"ID":"379363eb-5d66-41cf-989c-08cf4a2008ce","Type":"ContainerDied","Data":"bd9d577f6943004a5a41bb9a4a04f6a0e4485b6445fb84f6bd7dd293d0f885dc"}
Oct 30 21:30:25.731289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.731267    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"
Oct 30 21:30:25.736752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.735371    4918 scope.go:122] "RemoveContainer" containerID="527758fdb9e491eb2065b39fa20dfd9327f4f52cce258fdfd4fdf4c697e0ba37"
Oct 30 21:30:25.736752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.736533    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"
Oct 30 21:30:25.737099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.736814    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx" event={"ID":"43a51867-ecfe-43e9-b630-b621e9b4e428","Type":"ContainerDied","Data":"0d51eb87e23378d5a96597dc8d30e93c97614083569599c0c21b014a129ca06e"}
Oct 30 21:30:25.747907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.747885    4918 scope.go:122] "RemoveContainer" containerID="0ac1006643811c41a9dc2a9842f30fff54c990636929207a71d71d7733477591"
Oct 30 21:30:25.758488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.758472    4918 scope.go:122] "RemoveContainer" containerID="51d10139eda1b0c14ceea25ce8b226b55a6642139a34af0dbf6683027e9c98b6"
Oct 30 21:30:25.760315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.760291    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"]
Oct 30 21:30:25.771077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.771059    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-kwv29"]
Oct 30 21:30:25.832993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.832964    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"]
Oct 30 21:30:25.848863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.848828    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889930-6xbqx"]
Oct 30 21:30:25.918419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.911568    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"]
Oct 30 21:30:25.918839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.918689    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-06bb0f-29889930-msrqt"]
Oct 30 21:30:25.954485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.954443    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"]
Oct 30 21:30:25.983465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:25.971009    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889930-fjnlg"]
Oct 30 21:30:27.079508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:27.079479    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" path="/var/lib/kubelet/pods/0a9f2c2a-8aea-4ecf-941b-de85d3bc6418/volumes"
Oct 30 21:30:27.079991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:27.079977    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce" path="/var/lib/kubelet/pods/379363eb-5d66-41cf-989c-08cf4a2008ce/volumes"
Oct 30 21:30:27.080442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:27.080428    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428" path="/var/lib/kubelet/pods/43a51867-ecfe-43e9-b630-b621e9b4e428/volumes"
Oct 30 21:30:27.081150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:27.081137    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" path="/var/lib/kubelet/pods/e1225dbb-da72-4c3f-b13d-b63d71e3ec51/volumes"
Oct 30 21:30:27.929559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:27.929520    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj" podUID="312757a6-8779-4a73-8e59-389d4a7dbb91" containerName="payload-processing" containerID="cri-o://5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c" gracePeriod=25
Oct 30 21:30:28.227766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.227741    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"
Oct 30 21:30:28.303341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.303304    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/312757a6-8779-4a73-8e59-389d4a7dbb91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312757a6-8779-4a73-8e59-389d4a7dbb91-plugins-config-volume\") pod \"312757a6-8779-4a73-8e59-389d4a7dbb91\" (UID: \"312757a6-8779-4a73-8e59-389d4a7dbb91\") "
Oct 30 21:30:28.303673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.303651    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/312757a6-8779-4a73-8e59-389d4a7dbb91-kube-api-access-q2jw2\" (UniqueName: \"kubernetes.io/projected/312757a6-8779-4a73-8e59-389d4a7dbb91-kube-api-access-q2jw2\") pod \"312757a6-8779-4a73-8e59-389d4a7dbb91\" (UID: \"312757a6-8779-4a73-8e59-389d4a7dbb91\") "
Oct 30 21:30:28.303793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.303696    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312757a6-8779-4a73-8e59-389d4a7dbb91-plugins-config-volume" pod "312757a6-8779-4a73-8e59-389d4a7dbb91" (UID: "312757a6-8779-4a73-8e59-389d4a7dbb91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:28.304421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.304378    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/312757a6-8779-4a73-8e59-389d4a7dbb91-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:28.306248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.306222    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312757a6-8779-4a73-8e59-389d4a7dbb91-kube-api-access-q2jw2" pod "312757a6-8779-4a73-8e59-389d4a7dbb91" (UID: "312757a6-8779-4a73-8e59-389d4a7dbb91"). InnerVolumeSpecName "kube-api-access-q2jw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:28.410451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.406938    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2jw2\" (UniqueName: \"kubernetes.io/projected/312757a6-8779-4a73-8e59-389d4a7dbb91-kube-api-access-q2jw2\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:28.769444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.769376    4918 generic.go:356] "Generic (PLEG): container finished" podID="312757a6-8779-4a73-8e59-389d4a7dbb91" containerID="5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c" exitCode=0
Oct 30 21:30:28.769444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.769439    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"
Oct 30 21:30:28.769660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.769469    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj" event={"ID":"312757a6-8779-4a73-8e59-389d4a7dbb91","Type":"ContainerDied","Data":"5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c"}
Oct 30 21:30:28.769660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.769528    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj" event={"ID":"312757a6-8779-4a73-8e59-389d4a7dbb91","Type":"ContainerDied","Data":"d33ea1802c9c7ca2be6e218da63f658ea80f68a962bbfc1fe7e73e536f1fd2fc"}
Oct 30 21:30:28.769660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.769557    4918 scope.go:122] "RemoveContainer" containerID="5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c"
Oct 30 21:30:28.791526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.791501    4918 scope.go:122] "RemoveContainer" containerID="5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c"
Oct 30 21:30:28.791830 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:28.791806    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c\": container with ID starting with 5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c not found: ID does not exist" containerID="5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c"
Oct 30 21:30:28.791903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.791840    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c"} err="failed to get container status \"5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c\": rpc error: code = NotFound desc = could not find container \"5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c\": container with ID starting with 5dedeb4d212feadb589d85deaa3b35bf0b9b0e4bf6b60265dbb4ed2bdf79ab1c not found: ID does not exist"
Oct 30 21:30:28.810669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.810644    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"]
Oct 30 21:30:28.814901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:28.814877    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-46c98c-5dbd76b855-kq9cj"]
Oct 30 21:30:29.105831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:29.105722    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="312757a6-8779-4a73-8e59-389d4a7dbb91" path="/var/lib/kubelet/pods/312757a6-8779-4a73-8e59-389d4a7dbb91/volumes"
Oct 30 21:30:29.201840 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:29.201802    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:30:30.726414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:30.725546    4918 scope.go:122] "RemoveContainer" containerID="056e83beac021fe0b47246449bf8dadc2fe1c63c7a8a0ec585a10ffcc0ae0710"
Oct 30 21:30:30.795153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:30.795125    4918 scope.go:122] "RemoveContainer" containerID="0d8819a292b0267d5bfb7468925962fc59e13292deb7b5a2f7389694ed63e6ce"
Oct 30 21:30:30.809645 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:30.809622    4918 scope.go:122] "RemoveContainer" containerID="81d8034aed5f073d988f60797548c47371d531cf8f16dfe1b123b63e7ad8278c"
Oct 30 21:30:30.822176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:30.822159    4918 scope.go:122] "RemoveContainer" containerID="513fdcc54d121673b8b02902c566e2ca26b4497ed66ffd937364ee59b494b169"
Oct 30 21:30:30.831911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:30.831886    4918 scope.go:122] "RemoveContainer" containerID="fc024cbfb2644dbd23e853f4b5eae83c2f386e34336a864c4f947abae50667c0"
Oct 30 21:30:30.844767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:30.844744    4918 scope.go:122] "RemoveContainer" containerID="c48d29846cdd2e27508b5989f4cd33cf501f0f84db5645f9e1176bb551705eea"
Oct 30 21:30:30.854660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:30.854643    4918 scope.go:122] "RemoveContainer" containerID="884ffcce38e6304429c3c3fe983e8504bda55780952af447003bf123b687e1f4"
Oct 30 21:30:37.069273 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:37.069179    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"]
Oct 30 21:30:37.092230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:37.092199    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"]
Oct 30 21:30:37.092926 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:37.092898    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-87e5d7d0-nwmv6\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="c77f820a-500a-4201-9fbd-1d470b63971b" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e5d7d0-nwmv6"
Oct 30 21:30:39.084168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:39.084134    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c77f820a-500a-4201-9fbd-1d470b63971b" path="/var/lib/kubelet/pods/c77f820a-500a-4201-9fbd-1d470b63971b/volumes"
Oct 30 21:30:40.696605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:40.696556    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"]
Oct 30 21:30:40.697068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:40.696918    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerName="istio-proxy" containerID="cri-o://d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e" gracePeriod=30
Oct 30 21:30:41.277022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.276992    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7660-610104"]
Oct 30 21:30:41.277713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277698    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9fbca41f-798b-4b86-b103-47868642c624" containerName="cleanup"
Oct 30 21:30:41.277764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277715    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbca41f-798b-4b86-b103-47868642c624" containerName="cleanup"
Oct 30 21:30:41.277764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277726    4918 cpu_manager.go:414] "containerMap: removing container" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468" containerName="cleanup"
Oct 30 21:30:41.277764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277731    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468" containerName="cleanup"
Oct 30 21:30:41.277764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277741    4918 cpu_manager.go:414] "containerMap: removing container" podUID="778db27c-4f22-4317-9753-b84985424b0b" containerName="cleanup"
Oct 30 21:30:41.277764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277747    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778db27c-4f22-4317-9753-b84985424b0b" containerName="cleanup"
Oct 30 21:30:41.277764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277754    4918 cpu_manager.go:414] "containerMap: removing container" podUID="17420e84-c277-42b9-99d8-c139e86395df" containerName="cleanup"
Oct 30 21:30:41.277764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277760    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17420e84-c277-42b9-99d8-c139e86395df" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277770    4918 cpu_manager.go:414] "containerMap: removing container" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277775    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277781    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277786    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277803    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9fbca41f-798b-4b86-b103-47868642c624" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277808    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbca41f-798b-4b86-b103-47868642c624" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277816    4918 cpu_manager.go:414] "containerMap: removing container" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277820    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277827    4918 cpu_manager.go:414] "containerMap: removing container" podUID="8f43a597-0a8a-4c75-8123-d5450be93d21" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277831    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f43a597-0a8a-4c75-8123-d5450be93d21" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277842    4918 cpu_manager.go:414] "containerMap: removing container" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277846    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277851    4918 cpu_manager.go:414] "containerMap: removing container" podUID="17420e84-c277-42b9-99d8-c139e86395df" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277855    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17420e84-c277-42b9-99d8-c139e86395df" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277866    4918 cpu_manager.go:414] "containerMap: removing container" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277870    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277875    4918 cpu_manager.go:414] "containerMap: removing container" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277880    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277888    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277892    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277899    4918 cpu_manager.go:414] "containerMap: removing container" podUID="8f43a597-0a8a-4c75-8123-d5450be93d21" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277903    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f43a597-0a8a-4c75-8123-d5450be93d21" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277909    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277913    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277919    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277923    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277936    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277940    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277947    4918 cpu_manager.go:414] "containerMap: removing container" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277951    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277959    4918 cpu_manager.go:414] "containerMap: removing container" podUID="778db27c-4f22-4317-9753-b84985424b0b" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277963    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778db27c-4f22-4317-9753-b84985424b0b" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277968    4918 cpu_manager.go:414] "containerMap: removing container" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277972    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277981    4918 cpu_manager.go:414] "containerMap: removing container" podUID="312757a6-8779-4a73-8e59-389d4a7dbb91" containerName="payload-processing"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277985    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="312757a6-8779-4a73-8e59-389d4a7dbb91" containerName="payload-processing"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277992    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.277996    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" containerName="cleanup"
Oct 30 21:30:41.277984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278003    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278007    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278015    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278018    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278029    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278033    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278045    4918 cpu_manager.go:414] "containerMap: removing container" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278049    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278061    4918 cpu_manager.go:414] "containerMap: removing container" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278066    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278183    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278190    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278195    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbca41f-798b-4b86-b103-47868642c624" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278201    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278207    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f43a597-0a8a-4c75-8123-d5450be93d21" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278212    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278218    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="17420e84-c277-42b9-99d8-c139e86395df" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278223    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278227    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278234    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278240    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f43a597-0a8a-4c75-8123-d5450be93d21" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278245    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278249    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278255    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278261    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="379363eb-5d66-41cf-989c-08cf4a2008ce" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278266    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="778db27c-4f22-4317-9753-b84985424b0b" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278272    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="312757a6-8779-4a73-8e59-389d4a7dbb91" containerName="payload-processing"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278278    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278282    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278287    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278292    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278299    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278306    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278311    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278321    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278326    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="778db27c-4f22-4317-9753-b84985424b0b" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278332    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278338    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbca41f-798b-4b86-b103-47868642c624" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278344    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="778db27c-4f22-4317-9753-b84985424b0b" containerName="cleanup"
Oct 30 21:30:41.279167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.278351    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="17420e84-c277-42b9-99d8-c139e86395df" containerName="cleanup"
Oct 30 21:30:41.286293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.286270    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:41.286863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.286839    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-610104"]
Oct 30 21:30:41.290257 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.290235    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:30:41.290375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.290342    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:30:41.304606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.304583    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9lmpp\""
Oct 30 21:30:41.365343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.365316    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrtb\" (UniqueName: \"kubernetes.io/projected/e4673db9-1dba-4971-96f6-410fdbde18d4-kube-api-access-whrtb\") pod \"test-curl-7660-610104\" (UID: \"e4673db9-1dba-4971-96f6-410fdbde18d4\") " pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:41.467008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.466977    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrtb\" (UniqueName: \"kubernetes.io/projected/e4673db9-1dba-4971-96f6-410fdbde18d4-kube-api-access-whrtb\") pod \"test-curl-7660-610104\" (UID: \"e4673db9-1dba-4971-96f6-410fdbde18d4\") " pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:41.480006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.479974    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrtb\" (UniqueName: \"kubernetes.io/projected/e4673db9-1dba-4971-96f6-410fdbde18d4-kube-api-access-whrtb\") pod \"test-curl-7660-610104\" (UID: \"e4673db9-1dba-4971-96f6-410fdbde18d4\") " pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:41.618527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.613715    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:41.959881 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:30:41.959853    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4673db9_1dba_4971_96f6_410fdbde18d4.slice/crio-27e4a37edd0113bfa052809ee4bdf72d08d0f33f00afe8253adc2baed26e0482 WatchSource:0}: Error finding container 27e4a37edd0113bfa052809ee4bdf72d08d0f33f00afe8253adc2baed26e0482: Status 404 returned error can't find the container with id 27e4a37edd0113bfa052809ee4bdf72d08d0f33f00afe8253adc2baed26e0482
Oct 30 21:30:41.960774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.960750    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-610104"]
Oct 30 21:30:41.982103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:41.982071    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-610104" event={"ID":"e4673db9-1dba-4971-96f6-410fdbde18d4","Type":"ContainerStarted","Data":"27e4a37edd0113bfa052809ee4bdf72d08d0f33f00afe8253adc2baed26e0482"}
Oct 30 21:30:43.009417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.000501    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-610104" event={"ID":"e4673db9-1dba-4971-96f6-410fdbde18d4","Type":"ContainerStarted","Data":"1d82c846d770021c10bf7798c71fac6032c15e7559009f78716570ec54559634"}
Oct 30 21:30:43.023475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.021428    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7660-610104" podStartSLOduration=2.021406905 podStartE2EDuration="2.021406905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:43.019508911 +0000 UTC m=+1756.066068068" watchObservedRunningTime="2026-10-30 21:30:43.021406905 +0000 UTC m=+1756.067966042"
Oct 30 21:30:43.699376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.699342    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7660-610104"]
Oct 30 21:30:43.704027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.703999    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7660-610104"]
Oct 30 21:30:43.865552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.865526    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7660-84ad22"]
Oct 30 21:30:43.866255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866240    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869" containerName="cleanup"
Oct 30 21:30:43.866315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866257    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc27f4e0-65dd-4971-abee-816ac50c9869" containerName="cleanup"
Oct 30 21:30:43.866315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866287    4918 cpu_manager.go:414] "containerMap: removing container" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468" containerName="cleanup"
Oct 30 21:30:43.866315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866292    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468" containerName="cleanup"
Oct 30 21:30:43.866315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866304    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9fbca41f-798b-4b86-b103-47868642c624" containerName="cleanup"
Oct 30 21:30:43.866315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866308    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbca41f-798b-4b86-b103-47868642c624" containerName="cleanup"
Oct 30 21:30:43.866315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866316    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e4673db9-1dba-4971-96f6-410fdbde18d4" containerName="test-curl-7660-610104"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866321    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4673db9-1dba-4971-96f6-410fdbde18d4" containerName="test-curl-7660-610104"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866334    4918 cpu_manager.go:414] "containerMap: removing container" podUID="17420e84-c277-42b9-99d8-c139e86395df" containerName="cleanup"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866338    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17420e84-c277-42b9-99d8-c139e86395df" containerName="cleanup"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866348    4918 cpu_manager.go:414] "containerMap: removing container" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428" containerName="cleanup"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866353    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a51867-ecfe-43e9-b630-b621e9b4e428" containerName="cleanup"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866360    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e" containerName="cleanup"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866364    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e" containerName="cleanup"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866369    4918 cpu_manager.go:414] "containerMap: removing container" podUID="8f43a597-0a8a-4c75-8123-d5450be93d21" containerName="cleanup"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866374    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f43a597-0a8a-4c75-8123-d5450be93d21" containerName="cleanup"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866406    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57" containerName="cleanup"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866411    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68adc98-00aa-48da-8690-4ce89ca58a57" containerName="cleanup"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866420    4918 cpu_manager.go:414] "containerMap: removing container" podUID="778db27c-4f22-4317-9753-b84985424b0b" containerName="cleanup"
Oct 30 21:30:43.866512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866424    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778db27c-4f22-4317-9753-b84985424b0b" containerName="cleanup"
Oct 30 21:30:43.866936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866532    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea1b87b-1562-4d1e-af32-8078f1e62468" containerName="cleanup"
Oct 30 21:30:43.866936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866544    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="17420e84-c277-42b9-99d8-c139e86395df" containerName="cleanup"
Oct 30 21:30:43.866936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866551    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbca41f-798b-4b86-b103-47868642c624" containerName="cleanup"
Oct 30 21:30:43.866936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866557    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9f2c2a-8aea-4ecf-941b-de85d3bc6418" containerName="cleanup"
Oct 30 21:30:43.866936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866563    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4673db9-1dba-4971-96f6-410fdbde18d4" containerName="test-curl-7660-610104"
Oct 30 21:30:43.866936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866571    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6c38be-7c77-435c-a127-3d3fd054574e" containerName="cleanup"
Oct 30 21:30:43.866936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866578    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f43a597-0a8a-4c75-8123-d5450be93d21" containerName="cleanup"
Oct 30 21:30:43.866936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.866584    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1225dbb-da72-4c3f-b13d-b63d71e3ec51" containerName="cleanup"
Oct 30 21:30:43.871123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.871096    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-84ad22"
Oct 30 21:30:43.876450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:43.875985    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-84ad22"]
Oct 30 21:30:44.014967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:44.014935    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5kp\" (UniqueName: \"kubernetes.io/projected/b9a2f831-2936-4f4f-bc13-5e9dad2b0be6-kube-api-access-ts5kp\") pod \"test-curl-7660-84ad22\" (UID: \"b9a2f831-2936-4f4f-bc13-5e9dad2b0be6\") " pod="opendatahub/test-curl-7660-84ad22"
Oct 30 21:30:44.116471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:44.116385    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5kp\" (UniqueName: \"kubernetes.io/projected/b9a2f831-2936-4f4f-bc13-5e9dad2b0be6-kube-api-access-ts5kp\") pod \"test-curl-7660-84ad22\" (UID: \"b9a2f831-2936-4f4f-bc13-5e9dad2b0be6\") " pod="opendatahub/test-curl-7660-84ad22"
Oct 30 21:30:44.126215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:44.126190    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5kp\" (UniqueName: \"kubernetes.io/projected/b9a2f831-2936-4f4f-bc13-5e9dad2b0be6-kube-api-access-ts5kp\") pod \"test-curl-7660-84ad22\" (UID: \"b9a2f831-2936-4f4f-bc13-5e9dad2b0be6\") " pod="opendatahub/test-curl-7660-84ad22"
Oct 30 21:30:44.189982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:44.189893    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-84ad22"
Oct 30 21:30:44.503072 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:30:44.503031    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9a2f831_2936_4f4f_bc13_5e9dad2b0be6.slice/crio-70119aea8d364416ebbcf90211279958926b254fa5e1fb09dccb305cc4c785bc WatchSource:0}: Error finding container 70119aea8d364416ebbcf90211279958926b254fa5e1fb09dccb305cc4c785bc: Status 404 returned error can't find the container with id 70119aea8d364416ebbcf90211279958926b254fa5e1fb09dccb305cc4c785bc
Oct 30 21:30:44.504925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:44.504885    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-84ad22"]
Oct 30 21:30:45.021643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:45.021576    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7660-610104" podUID="e4673db9-1dba-4971-96f6-410fdbde18d4" containerName="test-curl-7660-610104" containerID="cri-o://1d82c846d770021c10bf7798c71fac6032c15e7559009f78716570ec54559634" gracePeriod=2
Oct 30 21:30:45.022529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:45.022505    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-84ad22" event={"ID":"b9a2f831-2936-4f4f-bc13-5e9dad2b0be6","Type":"ContainerStarted","Data":"53d1fafe139825ed16db0d668bd139c269dc359fb2eae5f081fbe263b5ad26ae"}
Oct 30 21:30:45.022643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:45.022539    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-84ad22" event={"ID":"b9a2f831-2936-4f4f-bc13-5e9dad2b0be6","Type":"ContainerStarted","Data":"70119aea8d364416ebbcf90211279958926b254fa5e1fb09dccb305cc4c785bc"}
Oct 30 21:30:45.027506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:45.027472    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"]
Oct 30 21:30:45.032135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:45.032113    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaacd451-z9kkb"]
Oct 30 21:30:45.041970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:45.041929    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7660-84ad22" podStartSLOduration=2.041912698 podStartE2EDuration="2.041912698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:45.040913649 +0000 UTC m=+1758.087472818" watchObservedRunningTime="2026-10-30 21:30:45.041912698 +0000 UTC m=+1758.088471843"
Oct 30 21:30:45.043547 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:45.043521    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-610104\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="e4673db9-1dba-4971-96f6-410fdbde18d4" pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:45.046059 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:45.046035    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-610104\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="e4673db9-1dba-4971-96f6-410fdbde18d4" pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:45.080762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:45.080734    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a1378c-4093-4ef9-ac01-a51049248f77" path="/var/lib/kubelet/pods/10a1378c-4093-4ef9-ac01-a51049248f77/volumes"
Oct 30 21:30:45.749305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:45.749271    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7660-84ad22"]
Oct 30 21:30:45.755472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:45.755443    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7660-84ad22"]
Oct 30 21:30:45.993676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:45.993650    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:30:46.012711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.012678    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7660-908ae7"]
Oct 30 21:30:46.013468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.013432    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b9a2f831-2936-4f4f-bc13-5e9dad2b0be6" containerName="test-curl-7660-84ad22"
Oct 30 21:30:46.013468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.013456    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a2f831-2936-4f4f-bc13-5e9dad2b0be6" containerName="test-curl-7660-84ad22"
Oct 30 21:30:46.013630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.013492    4918 cpu_manager.go:414] "containerMap: removing container" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerName="istio-proxy"
Oct 30 21:30:46.013630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.013499    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerName="istio-proxy"
Oct 30 21:30:46.013630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.013548    4918 cpu_manager.go:414] "containerMap: removing container" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerName="istio-proxy"
Oct 30 21:30:46.013630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.013555    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerName="istio-proxy"
Oct 30 21:30:46.013865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.013765    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a2f831-2936-4f4f-bc13-5e9dad2b0be6" containerName="test-curl-7660-84ad22"
Oct 30 21:30:46.013865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.013783    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerName="istio-proxy"
Oct 30 21:30:46.017851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.017784    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-908ae7"
Oct 30 21:30:46.035330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.035301    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-908ae7"]
Oct 30 21:30:46.035872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.035842    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" event={"ID":"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2","Type":"ContainerDied","Data":"d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e"}
Oct 30 21:30:46.035872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.035429    4918 generic.go:356] "Generic (PLEG): container finished" podID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerID="d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e" exitCode=0
Oct 30 21:30:46.036033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.035893    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv" event={"ID":"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2","Type":"ContainerDied","Data":"0261a26ca24bd34e0d6da146b81806c7ce44283c9b9c8bfb47a6db750bc94561"}
Oct 30 21:30:46.036033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.035923    4918 scope.go:122] "RemoveContainer" containerID="d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e"
Oct 30 21:30:46.036033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.035499    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"
Oct 30 21:30:46.048935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.048911    4918 scope.go:122] "RemoveContainer" containerID="4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80"
Oct 30 21:30:46.121569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.118038    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-credential-socket\") pod \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") "
Oct 30 21:30:46.128101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.123300    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-credential-socket" pod "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" (UID: "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:46.128101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.123658    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-data\") pod \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") "
Oct 30 21:30:46.128101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.127627    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-data" pod "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" (UID: "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:46.128101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.127731    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-certs\") pod \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") "
Oct 30 21:30:46.128613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.128586    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-certs" pod "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" (UID: "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:46.133658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.133627    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-token\" (UniqueName: \"kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-token\") pod \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") "
Oct 30 21:30:46.133844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.133723    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-envoy\") pod \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") "
Oct 30 21:30:46.134564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.134541    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istiod-ca-cert\") pod \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") "
Oct 30 21:30:46.144544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.135150    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istiod-ca-cert" pod "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" (UID: "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:30:46.144544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.135345    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-kube-api-access-hl4r8\" (UniqueName: \"kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-kube-api-access-hl4r8\") pod \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") "
Oct 30 21:30:46.144544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.136133    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-socket\") pod \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") "
Oct 30 21:30:46.144544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.136727    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-socket" pod "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" (UID: "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:46.144544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.136916    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-podinfo\") pod \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\" (UID: \"bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2\") "
Oct 30 21:30:46.144544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.139067    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7cnv\" (UniqueName: \"kubernetes.io/projected/6627012a-a6b5-487a-b273-bfbf3c23e7c3-kube-api-access-b7cnv\") pod \"test-curl-7660-908ae7\" (UID: \"6627012a-a6b5-487a-b273-bfbf3c23e7c3\") " pod="opendatahub/test-curl-7660-908ae7"
Oct 30 21:30:46.144544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.140357    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.144544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.141546    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.144544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.141566    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.144544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.141583    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.144544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.141601    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.173411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.169717    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-envoy" pod "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" (UID: "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:30:46.188878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.186644    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-kube-api-access-hl4r8" pod "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" (UID: "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2"). InnerVolumeSpecName "kube-api-access-hl4r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:46.195666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.192984    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-token" pod "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" (UID: "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:46.197518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.197488    4918 scope.go:122] "RemoveContainer" containerID="d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e"
Oct 30 21:30:46.199027 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:46.198968    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e\": container with ID starting with d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e not found: ID does not exist" containerID="d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e"
Oct 30 21:30:46.199027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.199009    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e"} err="failed to get container status \"d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e\": rpc error: code = NotFound desc = could not find container \"d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e\": container with ID starting with d8ffb5210bbe962d3208cc425189d32642779e8803d5cc26df9f571690b84d4e not found: ID does not exist"
Oct 30 21:30:46.199190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.199039    4918 scope.go:122] "RemoveContainer" containerID="4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80"
Oct 30 21:30:46.199497 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:46.199465    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80\": container with ID starting with 4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80 not found: ID does not exist" containerID="4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80"
Oct 30 21:30:46.199606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.199508    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80"} err="failed to get container status \"4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80\": rpc error: code = NotFound desc = could not find container \"4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80\": container with ID starting with 4a3d849f2ec7eb998c63a2cf458352e4d619650fdc2877d0cabe9c60b7a94a80 not found: ID does not exist"
Oct 30 21:30:46.201568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.201537    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-podinfo" pod "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" (UID: "bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:30:46.243267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.243225    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7cnv\" (UniqueName: \"kubernetes.io/projected/6627012a-a6b5-487a-b273-bfbf3c23e7c3-kube-api-access-b7cnv\") pod \"test-curl-7660-908ae7\" (UID: \"6627012a-a6b5-487a-b273-bfbf3c23e7c3\") " pod="opendatahub/test-curl-7660-908ae7"
Oct 30 21:30:46.243425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.243371    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl4r8\" (UniqueName: \"kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-kube-api-access-hl4r8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.243486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.243384    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.243577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.243494    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.243577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.243514    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:46.252311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.252284    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7cnv\" (UniqueName: \"kubernetes.io/projected/6627012a-a6b5-487a-b273-bfbf3c23e7c3-kube-api-access-b7cnv\") pod \"test-curl-7660-908ae7\" (UID: \"6627012a-a6b5-487a-b273-bfbf3c23e7c3\") " pod="opendatahub/test-curl-7660-908ae7"
Oct 30 21:30:46.335973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.335943    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-908ae7"
Oct 30 21:30:46.368993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.368969    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"]
Oct 30 21:30:46.373805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.373775    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-46c98c-openshift-default-5b49c9987c-d6kmv"]
Oct 30 21:30:46.623999 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:30:46.623971    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6627012a_a6b5_487a_b273_bfbf3c23e7c3.slice/crio-f0181a09f0a2aba7b0a95f738d9aaa7c4ccbd6cea5101ea7a3be37df2281ae48 WatchSource:0}: Error finding container f0181a09f0a2aba7b0a95f738d9aaa7c4ccbd6cea5101ea7a3be37df2281ae48: Status 404 returned error can't find the container with id f0181a09f0a2aba7b0a95f738d9aaa7c4ccbd6cea5101ea7a3be37df2281ae48
Oct 30 21:30:46.625482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:46.625461    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-908ae7"]
Oct 30 21:30:47.049428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:47.049381    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-908ae7" event={"ID":"6627012a-a6b5-487a-b273-bfbf3c23e7c3","Type":"ContainerStarted","Data":"e584353d697181f904c8a71584456ceade70f2bb904210284f90e42f874fd294"}
Oct 30 21:30:47.049740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:47.049446    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-908ae7" event={"ID":"6627012a-a6b5-487a-b273-bfbf3c23e7c3","Type":"ContainerStarted","Data":"f0181a09f0a2aba7b0a95f738d9aaa7c4ccbd6cea5101ea7a3be37df2281ae48"}
Oct 30 21:30:47.049740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:47.049498    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7660-84ad22" podUID="b9a2f831-2936-4f4f-bc13-5e9dad2b0be6" containerName="test-curl-7660-84ad22" containerID="cri-o://53d1fafe139825ed16db0d668bd139c269dc359fb2eae5f081fbe263b5ad26ae" gracePeriod=2
Oct 30 21:30:47.066514 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:47.066484    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-84ad22\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="b9a2f831-2936-4f4f-bc13-5e9dad2b0be6" pod="opendatahub/test-curl-7660-84ad22"
Oct 30 21:30:47.069102 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:47.069063    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-84ad22\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="b9a2f831-2936-4f4f-bc13-5e9dad2b0be6" pod="opendatahub/test-curl-7660-84ad22"
Oct 30 21:30:47.069611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:47.069563    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7660-908ae7" podStartSLOduration=2.069548056 podStartE2EDuration="2.069548056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:47.064101534 +0000 UTC m=+1760.110660702" watchObservedRunningTime="2026-10-30 21:30:47.069548056 +0000 UTC m=+1760.116107204"
Oct 30 21:30:47.077126 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:47.077099    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-84ad22\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="b9a2f831-2936-4f4f-bc13-5e9dad2b0be6" pod="opendatahub/test-curl-7660-84ad22"
Oct 30 21:30:47.083196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:47.083174    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" path="/var/lib/kubelet/pods/bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2/volumes"
Oct 30 21:30:47.245841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:47.245817    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:47.248763 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:47.248734    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-610104\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="e4673db9-1dba-4971-96f6-410fdbde18d4" pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:47.261502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:47.261480    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4673db9-1dba-4971-96f6-410fdbde18d4-kube-api-access-whrtb\" (UniqueName: \"kubernetes.io/projected/e4673db9-1dba-4971-96f6-410fdbde18d4-kube-api-access-whrtb\") pod \"e4673db9-1dba-4971-96f6-410fdbde18d4\" (UID: \"e4673db9-1dba-4971-96f6-410fdbde18d4\") "
Oct 30 21:30:47.263674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:47.263655    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4673db9-1dba-4971-96f6-410fdbde18d4-kube-api-access-whrtb" pod "e4673db9-1dba-4971-96f6-410fdbde18d4" (UID: "e4673db9-1dba-4971-96f6-410fdbde18d4"). InnerVolumeSpecName "kube-api-access-whrtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:47.363260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:47.363183    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whrtb\" (UniqueName: \"kubernetes.io/projected/e4673db9-1dba-4971-96f6-410fdbde18d4-kube-api-access-whrtb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:47.860701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:47.860670    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7660-908ae7"]
Oct 30 21:30:47.867402 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:47.867364    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7660-908ae7"]
Oct 30 21:30:48.026079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.026047    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7660-9ffec5"]
Oct 30 21:30:48.026864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.026846    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6627012a-a6b5-487a-b273-bfbf3c23e7c3" containerName="test-curl-7660-908ae7"
Oct 30 21:30:48.026925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.026866    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6627012a-a6b5-487a-b273-bfbf3c23e7c3" containerName="test-curl-7660-908ae7"
Oct 30 21:30:48.027010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.027001    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb3b40d-2f16-4bc1-b0be-3dbbc759e6e2" containerName="istio-proxy"
Oct 30 21:30:48.027047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.027014    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6627012a-a6b5-487a-b273-bfbf3c23e7c3" containerName="test-curl-7660-908ae7"
Oct 30 21:30:48.030523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.030504    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-9ffec5"
Oct 30 21:30:48.033689 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:48.033643    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-610104\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="e4673db9-1dba-4971-96f6-410fdbde18d4" pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:48.036786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.036765    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-9ffec5"]
Oct 30 21:30:48.064407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.064370    4918 generic.go:356] "Generic (PLEG): container finished" podID="e4673db9-1dba-4971-96f6-410fdbde18d4" containerID="1d82c846d770021c10bf7798c71fac6032c15e7559009f78716570ec54559634" exitCode=137
Oct 30 21:30:48.064839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.064434    4918 scope.go:122] "RemoveContainer" containerID="1d82c846d770021c10bf7798c71fac6032c15e7559009f78716570ec54559634"
Oct 30 21:30:48.064839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.064462    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:48.067598 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:48.067550    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-610104\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="e4673db9-1dba-4971-96f6-410fdbde18d4" pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:48.076727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.076710    4918 scope.go:122] "RemoveContainer" containerID="1d82c846d770021c10bf7798c71fac6032c15e7559009f78716570ec54559634"
Oct 30 21:30:48.076970 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:48.076932    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-610104\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="e4673db9-1dba-4971-96f6-410fdbde18d4" pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:48.077098 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:48.077017    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d82c846d770021c10bf7798c71fac6032c15e7559009f78716570ec54559634\": container with ID starting with 1d82c846d770021c10bf7798c71fac6032c15e7559009f78716570ec54559634 not found: ID does not exist" containerID="1d82c846d770021c10bf7798c71fac6032c15e7559009f78716570ec54559634"
Oct 30 21:30:48.077098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.077053    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d82c846d770021c10bf7798c71fac6032c15e7559009f78716570ec54559634"} err="failed to get container status \"1d82c846d770021c10bf7798c71fac6032c15e7559009f78716570ec54559634\": rpc error: code = NotFound desc = could not find container \"1d82c846d770021c10bf7798c71fac6032c15e7559009f78716570ec54559634\": container with ID starting with 1d82c846d770021c10bf7798c71fac6032c15e7559009f78716570ec54559634 not found: ID does not exist"
Oct 30 21:30:48.096709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.096682    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57dzd\" (UniqueName: \"kubernetes.io/projected/a752f921-d844-4c8e-813e-1064c9b74a25-kube-api-access-57dzd\") pod \"test-curl-7660-9ffec5\" (UID: \"a752f921-d844-4c8e-813e-1064c9b74a25\") " pod="opendatahub/test-curl-7660-9ffec5"
Oct 30 21:30:48.198672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.198554    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57dzd\" (UniqueName: \"kubernetes.io/projected/a752f921-d844-4c8e-813e-1064c9b74a25-kube-api-access-57dzd\") pod \"test-curl-7660-9ffec5\" (UID: \"a752f921-d844-4c8e-813e-1064c9b74a25\") " pod="opendatahub/test-curl-7660-9ffec5"
Oct 30 21:30:48.207537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.207512    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57dzd\" (UniqueName: \"kubernetes.io/projected/a752f921-d844-4c8e-813e-1064c9b74a25-kube-api-access-57dzd\") pod \"test-curl-7660-9ffec5\" (UID: \"a752f921-d844-4c8e-813e-1064c9b74a25\") " pod="opendatahub/test-curl-7660-9ffec5"
Oct 30 21:30:48.347178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.347143    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-9ffec5"
Oct 30 21:30:48.731047 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:30:48.731013    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda752f921_d844_4c8e_813e_1064c9b74a25.slice/crio-4937c566d9cffe114df77cabcb5d6b4e74815fd537e1514d82558b755638892b WatchSource:0}: Error finding container 4937c566d9cffe114df77cabcb5d6b4e74815fd537e1514d82558b755638892b: Status 404 returned error can't find the container with id 4937c566d9cffe114df77cabcb5d6b4e74815fd537e1514d82558b755638892b
Oct 30 21:30:48.731762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:48.731738    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-9ffec5"]
Oct 30 21:30:49.076284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:49.076249    4918 generic.go:356] "Generic (PLEG): container finished" podID="b9a2f831-2936-4f4f-bc13-5e9dad2b0be6" containerID="53d1fafe139825ed16db0d668bd139c269dc359fb2eae5f081fbe263b5ad26ae" exitCode=137
Oct 30 21:30:49.078743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:49.078713    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7660-908ae7" podUID="6627012a-a6b5-487a-b273-bfbf3c23e7c3" containerName="test-curl-7660-908ae7" containerID="cri-o://e584353d697181f904c8a71584456ceade70f2bb904210284f90e42f874fd294" gracePeriod=2
Oct 30 21:30:49.081892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:49.081868    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4673db9-1dba-4971-96f6-410fdbde18d4" path="/var/lib/kubelet/pods/e4673db9-1dba-4971-96f6-410fdbde18d4/volumes"
Oct 30 21:30:49.082218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:49.082201    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-9ffec5" event={"ID":"a752f921-d844-4c8e-813e-1064c9b74a25","Type":"ContainerStarted","Data":"d82abf7dababc3a309aabb99be6bf57fc17034e9dccd32ad38a398814886ffe5"}
Oct 30 21:30:49.082283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:49.082235    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-9ffec5" event={"ID":"a752f921-d844-4c8e-813e-1064c9b74a25","Type":"ContainerStarted","Data":"4937c566d9cffe114df77cabcb5d6b4e74815fd537e1514d82558b755638892b"}
Oct 30 21:30:49.083801 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:49.083775    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-610104\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="e4673db9-1dba-4971-96f6-410fdbde18d4" pod="opendatahub/test-curl-7660-610104"
Oct 30 21:30:49.134855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:49.134788    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7660-9ffec5" podStartSLOduration=1.134770924 podStartE2EDuration="1.134770924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:49.132912449 +0000 UTC m=+1762.179471593" watchObservedRunningTime="2026-10-30 21:30:49.134770924 +0000 UTC m=+1762.181330072"
Oct 30 21:30:49.260808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:49.260784    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-84ad22"
Oct 30 21:30:49.321892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:49.321817    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a2f831-2936-4f4f-bc13-5e9dad2b0be6-kube-api-access-ts5kp\" (UniqueName: \"kubernetes.io/projected/b9a2f831-2936-4f4f-bc13-5e9dad2b0be6-kube-api-access-ts5kp\") pod \"b9a2f831-2936-4f4f-bc13-5e9dad2b0be6\" (UID: \"b9a2f831-2936-4f4f-bc13-5e9dad2b0be6\") "
Oct 30 21:30:49.324164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:49.324137    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a2f831-2936-4f4f-bc13-5e9dad2b0be6-kube-api-access-ts5kp" pod "b9a2f831-2936-4f4f-bc13-5e9dad2b0be6" (UID: "b9a2f831-2936-4f4f-bc13-5e9dad2b0be6"). InnerVolumeSpecName "kube-api-access-ts5kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:49.423946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:49.423910    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts5kp\" (UniqueName: \"kubernetes.io/projected/b9a2f831-2936-4f4f-bc13-5e9dad2b0be6-kube-api-access-ts5kp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:49.913297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:49.913255    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7660-9ffec5"]
Oct 30 21:30:49.932125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:49.932091    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7660-9ffec5"]
Oct 30 21:30:50.138422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.137635    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-84ad22"
Oct 30 21:30:50.138422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.138088    4918 scope.go:122] "RemoveContainer" containerID="53d1fafe139825ed16db0d668bd139c269dc359fb2eae5f081fbe263b5ad26ae"
Oct 30 21:30:50.146972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.141084    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7660-267ee2"]
Oct 30 21:30:50.146972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.142331    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a752f921-d844-4c8e-813e-1064c9b74a25" containerName="test-curl-7660-9ffec5"
Oct 30 21:30:50.146972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.142354    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a752f921-d844-4c8e-813e-1064c9b74a25" containerName="test-curl-7660-9ffec5"
Oct 30 21:30:50.146972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.142629    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a752f921-d844-4c8e-813e-1064c9b74a25" containerName="test-curl-7660-9ffec5"
Oct 30 21:30:50.152379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.152352    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-267ee2"]
Oct 30 21:30:50.152666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.152649    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-267ee2"
Oct 30 21:30:50.153802 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:50.153762    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-84ad22\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="b9a2f831-2936-4f4f-bc13-5e9dad2b0be6" pod="opendatahub/test-curl-7660-84ad22"
Oct 30 21:30:50.157592 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:50.157457    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-84ad22\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="b9a2f831-2936-4f4f-bc13-5e9dad2b0be6" pod="opendatahub/test-curl-7660-84ad22"
Oct 30 21:30:50.215748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.215673    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6h2k\" (UniqueName: \"kubernetes.io/projected/6df2bdd5-bea6-41c5-9013-a5f09a2d726f-kube-api-access-g6h2k\") pod \"test-curl-7660-267ee2\" (UID: \"6df2bdd5-bea6-41c5-9013-a5f09a2d726f\") " pod="opendatahub/test-curl-7660-267ee2"
Oct 30 21:30:50.317777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.317743    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6h2k\" (UniqueName: \"kubernetes.io/projected/6df2bdd5-bea6-41c5-9013-a5f09a2d726f-kube-api-access-g6h2k\") pod \"test-curl-7660-267ee2\" (UID: \"6df2bdd5-bea6-41c5-9013-a5f09a2d726f\") " pod="opendatahub/test-curl-7660-267ee2"
Oct 30 21:30:50.330715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.330693    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6h2k\" (UniqueName: \"kubernetes.io/projected/6df2bdd5-bea6-41c5-9013-a5f09a2d726f-kube-api-access-g6h2k\") pod \"test-curl-7660-267ee2\" (UID: \"6df2bdd5-bea6-41c5-9013-a5f09a2d726f\") " pod="opendatahub/test-curl-7660-267ee2"
Oct 30 21:30:50.477800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.477747    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-267ee2"
Oct 30 21:30:50.818981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:50.818955    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-267ee2"]
Oct 30 21:30:50.819722 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:30:50.819699    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6df2bdd5_bea6_41c5_9013_a5f09a2d726f.slice/crio-8dff3fb25a7679b6f873ef4f34dfdfea89eb63b17f7f9a6354161cbaaba0a9b0 WatchSource:0}: Error finding container 8dff3fb25a7679b6f873ef4f34dfdfea89eb63b17f7f9a6354161cbaaba0a9b0: Status 404 returned error can't find the container with id 8dff3fb25a7679b6f873ef4f34dfdfea89eb63b17f7f9a6354161cbaaba0a9b0
Oct 30 21:30:51.029242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.029204    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"]
Oct 30 21:30:51.033407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.033371    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4be7be0-8mx4h"]
Oct 30 21:30:51.080144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.080070    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b89002a6-095a-4945-b27d-aa0a51f6fb65" path="/var/lib/kubelet/pods/b89002a6-095a-4945-b27d-aa0a51f6fb65/volumes"
Oct 30 21:30:51.080574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.080558    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a2f831-2936-4f4f-bc13-5e9dad2b0be6" path="/var/lib/kubelet/pods/b9a2f831-2936-4f4f-bc13-5e9dad2b0be6/volumes"
Oct 30 21:30:51.149933 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.149895    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-267ee2" event={"ID":"6df2bdd5-bea6-41c5-9013-a5f09a2d726f","Type":"ContainerStarted","Data":"931d3e04eb8f02c63962bbad696218990b9d36e53d17ff62f74a473215f0f816"}
Oct 30 21:30:51.150482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.149947    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-267ee2" event={"ID":"6df2bdd5-bea6-41c5-9013-a5f09a2d726f","Type":"ContainerStarted","Data":"8dff3fb25a7679b6f873ef4f34dfdfea89eb63b17f7f9a6354161cbaaba0a9b0"}
Oct 30 21:30:51.151772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.151746    4918 generic.go:356] "Generic (PLEG): container finished" podID="6627012a-a6b5-487a-b273-bfbf3c23e7c3" containerID="e584353d697181f904c8a71584456ceade70f2bb904210284f90e42f874fd294" exitCode=137
Oct 30 21:30:51.154072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.154040    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7660-9ffec5" podUID="a752f921-d844-4c8e-813e-1064c9b74a25" containerName="test-curl-7660-9ffec5" containerID="cri-o://d82abf7dababc3a309aabb99be6bf57fc17034e9dccd32ad38a398814886ffe5" gracePeriod=2
Oct 30 21:30:51.174628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.173700    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7660-267ee2" podStartSLOduration=1.173682447 podStartE2EDuration="1.173682447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:51.16862809 +0000 UTC m=+1764.215187261" watchObservedRunningTime="2026-10-30 21:30:51.173682447 +0000 UTC m=+1764.220241593"
Oct 30 21:30:51.248887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.248860    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-908ae7"
Oct 30 21:30:51.355383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.355355    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6627012a-a6b5-487a-b273-bfbf3c23e7c3-kube-api-access-b7cnv\" (UniqueName: \"kubernetes.io/projected/6627012a-a6b5-487a-b273-bfbf3c23e7c3-kube-api-access-b7cnv\") pod \"6627012a-a6b5-487a-b273-bfbf3c23e7c3\" (UID: \"6627012a-a6b5-487a-b273-bfbf3c23e7c3\") "
Oct 30 21:30:51.357631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.357601    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6627012a-a6b5-487a-b273-bfbf3c23e7c3-kube-api-access-b7cnv" pod "6627012a-a6b5-487a-b273-bfbf3c23e7c3" (UID: "6627012a-a6b5-487a-b273-bfbf3c23e7c3"). InnerVolumeSpecName "kube-api-access-b7cnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:51.458590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.457792    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7cnv\" (UniqueName: \"kubernetes.io/projected/6627012a-a6b5-487a-b273-bfbf3c23e7c3-kube-api-access-b7cnv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:51.902408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.902359    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7660-267ee2"]
Oct 30 21:30:51.906501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:51.906476    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7660-267ee2"]
Oct 30 21:30:52.151422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.146920    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7660-790016"]
Oct 30 21:30:52.179726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.171927    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6df2bdd5-bea6-41c5-9013-a5f09a2d726f" containerName="test-curl-7660-267ee2"
Oct 30 21:30:52.179726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.171960    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df2bdd5-bea6-41c5-9013-a5f09a2d726f" containerName="test-curl-7660-267ee2"
Oct 30 21:30:52.186426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.180943    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df2bdd5-bea6-41c5-9013-a5f09a2d726f" containerName="test-curl-7660-267ee2"
Oct 30 21:30:52.204515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.204482    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-908ae7"
Oct 30 21:30:52.205698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.205672    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-790016"]
Oct 30 21:30:52.205829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.205740    4918 scope.go:122] "RemoveContainer" containerID="e584353d697181f904c8a71584456ceade70f2bb904210284f90e42f874fd294"
Oct 30 21:30:52.205829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.205802    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-790016"
Oct 30 21:30:52.225487 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:52.225454    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-908ae7\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="6627012a-a6b5-487a-b273-bfbf3c23e7c3" pod="opendatahub/test-curl-7660-908ae7"
Oct 30 21:30:52.295698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.295659    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhpc\" (UniqueName: \"kubernetes.io/projected/9cb6aed9-10f1-4a0d-a95a-b4adb46a1356-kube-api-access-srhpc\") pod \"test-curl-7660-790016\" (UID: \"9cb6aed9-10f1-4a0d-a95a-b4adb46a1356\") " pod="opendatahub/test-curl-7660-790016"
Oct 30 21:30:52.397591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.397553    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhpc\" (UniqueName: \"kubernetes.io/projected/9cb6aed9-10f1-4a0d-a95a-b4adb46a1356-kube-api-access-srhpc\") pod \"test-curl-7660-790016\" (UID: \"9cb6aed9-10f1-4a0d-a95a-b4adb46a1356\") " pod="opendatahub/test-curl-7660-790016"
Oct 30 21:30:52.406278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.406253    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhpc\" (UniqueName: \"kubernetes.io/projected/9cb6aed9-10f1-4a0d-a95a-b4adb46a1356-kube-api-access-srhpc\") pod \"test-curl-7660-790016\" (UID: \"9cb6aed9-10f1-4a0d-a95a-b4adb46a1356\") " pod="opendatahub/test-curl-7660-790016"
Oct 30 21:30:52.525719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.525682    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-790016"
Oct 30 21:30:52.855751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:52.855720    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-790016"]
Oct 30 21:30:52.856112 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:30:52.856082    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cb6aed9_10f1_4a0d_a95a_b4adb46a1356.slice/crio-e111749acb631f786dc4cd3bbdde2ca55390d8d749672bd32a2390190b307a26 WatchSource:0}: Error finding container e111749acb631f786dc4cd3bbdde2ca55390d8d749672bd32a2390190b307a26: Status 404 returned error can't find the container with id e111749acb631f786dc4cd3bbdde2ca55390d8d749672bd32a2390190b307a26
Oct 30 21:30:53.079579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:53.079548    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6627012a-a6b5-487a-b273-bfbf3c23e7c3" path="/var/lib/kubelet/pods/6627012a-a6b5-487a-b273-bfbf3c23e7c3/volumes"
Oct 30 21:30:53.214425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:53.214377    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-790016" event={"ID":"9cb6aed9-10f1-4a0d-a95a-b4adb46a1356","Type":"ContainerStarted","Data":"2dafb4b6057861248e5e97047c86f5b5686f952359bca5f0003553936a476823"}
Oct 30 21:30:53.214818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:53.214440    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-790016" event={"ID":"9cb6aed9-10f1-4a0d-a95a-b4adb46a1356","Type":"ContainerStarted","Data":"e111749acb631f786dc4cd3bbdde2ca55390d8d749672bd32a2390190b307a26"}
Oct 30 21:30:53.217343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:53.217318    4918 generic.go:356] "Generic (PLEG): container finished" podID="a752f921-d844-4c8e-813e-1064c9b74a25" containerID="d82abf7dababc3a309aabb99be6bf57fc17034e9dccd32ad38a398814886ffe5" exitCode=137
Oct 30 21:30:53.217670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:53.217626    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7660-267ee2" podUID="6df2bdd5-bea6-41c5-9013-a5f09a2d726f" containerName="test-curl-7660-267ee2" containerID="cri-o://931d3e04eb8f02c63962bbad696218990b9d36e53d17ff62f74a473215f0f816" gracePeriod=2
Oct 30 21:30:53.339302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:53.339279    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-9ffec5"
Oct 30 21:30:53.440307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:53.440279    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a752f921-d844-4c8e-813e-1064c9b74a25-kube-api-access-57dzd\" (UniqueName: \"kubernetes.io/projected/a752f921-d844-4c8e-813e-1064c9b74a25-kube-api-access-57dzd\") pod \"a752f921-d844-4c8e-813e-1064c9b74a25\" (UID: \"a752f921-d844-4c8e-813e-1064c9b74a25\") "
Oct 30 21:30:53.442646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:53.442620    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a752f921-d844-4c8e-813e-1064c9b74a25-kube-api-access-57dzd" pod "a752f921-d844-4c8e-813e-1064c9b74a25" (UID: "a752f921-d844-4c8e-813e-1064c9b74a25"). InnerVolumeSpecName "kube-api-access-57dzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:53.550415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:53.543347    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57dzd\" (UniqueName: \"kubernetes.io/projected/a752f921-d844-4c8e-813e-1064c9b74a25-kube-api-access-57dzd\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:53.917266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:53.917213    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7660-790016" podStartSLOduration=1.91719901 podStartE2EDuration="1.91719901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:53.232541675 +0000 UTC m=+1766.279100820" watchObservedRunningTime="2026-10-30 21:30:53.91719901 +0000 UTC m=+1766.963758155"
Oct 30 21:30:53.918928 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:53.918890    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7660-790016"]
Oct 30 21:30:53.924498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:53.924473    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7660-790016"]
Oct 30 21:30:54.227997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:54.227976    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-9ffec5"
Oct 30 21:30:54.228454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:54.228021    4918 scope.go:122] "RemoveContainer" containerID="d82abf7dababc3a309aabb99be6bf57fc17034e9dccd32ad38a398814886ffe5"
Oct 30 21:30:54.240456 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:54.240429    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-9ffec5\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="a752f921-d844-4c8e-813e-1064c9b74a25" pod="opendatahub/test-curl-7660-9ffec5"
Oct 30 21:30:55.135447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:55.134460    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a752f921-d844-4c8e-813e-1064c9b74a25" path="/var/lib/kubelet/pods/a752f921-d844-4c8e-813e-1064c9b74a25/volumes"
Oct 30 21:30:55.237844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:55.237808    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7660-790016" podUID="9cb6aed9-10f1-4a0d-a95a-b4adb46a1356" containerName="test-curl-7660-790016" containerID="cri-o://2dafb4b6057861248e5e97047c86f5b5686f952359bca5f0003553936a476823" gracePeriod=2
Oct 30 21:30:55.440511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:55.440481    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-267ee2"
Oct 30 21:30:55.607371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:55.607340    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df2bdd5-bea6-41c5-9013-a5f09a2d726f-kube-api-access-g6h2k\" (UniqueName: \"kubernetes.io/projected/6df2bdd5-bea6-41c5-9013-a5f09a2d726f-kube-api-access-g6h2k\") pod \"6df2bdd5-bea6-41c5-9013-a5f09a2d726f\" (UID: \"6df2bdd5-bea6-41c5-9013-a5f09a2d726f\") "
Oct 30 21:30:55.609595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:55.609562    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df2bdd5-bea6-41c5-9013-a5f09a2d726f-kube-api-access-g6h2k" pod "6df2bdd5-bea6-41c5-9013-a5f09a2d726f" (UID: "6df2bdd5-bea6-41c5-9013-a5f09a2d726f"). InnerVolumeSpecName "kube-api-access-g6h2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:55.710042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:55.709966    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6h2k\" (UniqueName: \"kubernetes.io/projected/6df2bdd5-bea6-41c5-9013-a5f09a2d726f-kube-api-access-g6h2k\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:56.191120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.191087    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"]
Oct 30 21:30:56.191964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.191939    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9cb6aed9-10f1-4a0d-a95a-b4adb46a1356" containerName="test-curl-7660-790016"
Oct 30 21:30:56.191964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.191963    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb6aed9-10f1-4a0d-a95a-b4adb46a1356" containerName="test-curl-7660-790016"
Oct 30 21:30:56.192221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.192206    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb6aed9-10f1-4a0d-a95a-b4adb46a1356" containerName="test-curl-7660-790016"
Oct 30 21:30:56.196127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.196102    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.205767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.204804    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e596df-openshift-default-dockercfg-tlppq\""
Oct 30 21:30:56.211088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.211061    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"]
Oct 30 21:30:56.250704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.250670    4918 generic.go:356] "Generic (PLEG): container finished" podID="6df2bdd5-bea6-41c5-9013-a5f09a2d726f" containerID="931d3e04eb8f02c63962bbad696218990b9d36e53d17ff62f74a473215f0f816" exitCode=137
Oct 30 21:30:56.251135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.250736    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-267ee2"
Oct 30 21:30:56.251135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.250763    4918 scope.go:122] "RemoveContainer" containerID="931d3e04eb8f02c63962bbad696218990b9d36e53d17ff62f74a473215f0f816"
Oct 30 21:30:56.271340 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:56.271295    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-267ee2\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="6df2bdd5-bea6-41c5-9013-a5f09a2d726f" pod="opendatahub/test-curl-7660-267ee2"
Oct 30 21:30:56.351415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.334687    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmfvh\" (UniqueName: \"kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-kube-api-access-rmfvh\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.370417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.361538    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-certs\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.370417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.361745    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istiod-ca-cert\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.370417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.361786    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-podinfo\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.370417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.361897    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-envoy\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.370417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.361967    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-credential-socket\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.370417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.362004    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-data\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.370417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.362063    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-socket\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.370417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.362092    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-token\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.483559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.483525    4918 scope.go:122] "RemoveContainer" containerID="931d3e04eb8f02c63962bbad696218990b9d36e53d17ff62f74a473215f0f816"
Oct 30 21:30:56.484992 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:56.484964    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931d3e04eb8f02c63962bbad696218990b9d36e53d17ff62f74a473215f0f816\": container with ID starting with 931d3e04eb8f02c63962bbad696218990b9d36e53d17ff62f74a473215f0f816 not found: ID does not exist" containerID="931d3e04eb8f02c63962bbad696218990b9d36e53d17ff62f74a473215f0f816"
Oct 30 21:30:56.485184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.485158    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931d3e04eb8f02c63962bbad696218990b9d36e53d17ff62f74a473215f0f816"} err="failed to get container status \"931d3e04eb8f02c63962bbad696218990b9d36e53d17ff62f74a473215f0f816\": rpc error: code = NotFound desc = could not find container \"931d3e04eb8f02c63962bbad696218990b9d36e53d17ff62f74a473215f0f816\": container with ID starting with 931d3e04eb8f02c63962bbad696218990b9d36e53d17ff62f74a473215f0f816 not found: ID does not exist"
Oct 30 21:30:56.502429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.498478    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-envoy\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.502429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.498555    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-credential-socket\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.502429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.498590    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-data\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.502429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.498636    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-socket\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.502429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.498661    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-token\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.502429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.498881    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmfvh\" (UniqueName: \"kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-kube-api-access-rmfvh\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.502429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.498918    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-certs\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.502429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.499022    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istiod-ca-cert\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.502429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.499049    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-podinfo\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.503050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.502847    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-socket\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.531417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.504271    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-credential-socket\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.531717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.505812    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-data\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.531862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.506045    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-certs\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.531946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.506758    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istiod-ca-cert\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.606501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.604983    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-podinfo\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.606501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.605288    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-token\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.628863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.620247    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-envoy\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.628863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.623683    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmfvh\" (UniqueName: \"kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-kube-api-access-rmfvh\") pod \"e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") " pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:56.817992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:56.817900    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:57.083331 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:57.083218    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-267ee2\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="6df2bdd5-bea6-41c5-9013-a5f09a2d726f" pod="opendatahub/test-curl-7660-267ee2"
Oct 30 21:30:57.089083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.089049    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df2bdd5-bea6-41c5-9013-a5f09a2d726f" path="/var/lib/kubelet/pods/6df2bdd5-bea6-41c5-9013-a5f09a2d726f/volumes"
Oct 30 21:30:57.096289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.096263    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"]
Oct 30 21:30:57.100430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.100358    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440077d5-xvjfp"]
Oct 30 21:30:57.114893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.114867    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"]
Oct 30 21:30:57.116613 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:30:57.116589    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7885c4_7a7c_4ba2_bc4d_0ae3fda7abb3.slice/crio-347ae8297144a67cd22c72aea0563d514d09e3d952779c2e8bdeb49c96257bff WatchSource:0}: Error finding container 347ae8297144a67cd22c72aea0563d514d09e3d952779c2e8bdeb49c96257bff: Status 404 returned error can't find the container with id 347ae8297144a67cd22c72aea0563d514d09e3d952779c2e8bdeb49c96257bff
Oct 30 21:30:57.119470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.119438    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:30:57.119583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.119510    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:30:57.119583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.119557    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:30:57.262557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.262518    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" event={"ID":"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3","Type":"ContainerStarted","Data":"3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03"}
Oct 30 21:30:57.262999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.262591    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" event={"ID":"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3","Type":"ContainerStarted","Data":"347ae8297144a67cd22c72aea0563d514d09e3d952779c2e8bdeb49c96257bff"}
Oct 30 21:30:57.291225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.291173    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" podStartSLOduration=1.291158114 podStartE2EDuration="1.291158114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:57.285559262 +0000 UTC m=+1770.332118432" watchObservedRunningTime="2026-10-30 21:30:57.291158114 +0000 UTC m=+1770.337717259"
Oct 30 21:30:57.601885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.601853    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-790016"
Oct 30 21:30:57.693422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.693316    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb6aed9-10f1-4a0d-a95a-b4adb46a1356-kube-api-access-srhpc\" (UniqueName: \"kubernetes.io/projected/9cb6aed9-10f1-4a0d-a95a-b4adb46a1356-kube-api-access-srhpc\") pod \"9cb6aed9-10f1-4a0d-a95a-b4adb46a1356\" (UID: \"9cb6aed9-10f1-4a0d-a95a-b4adb46a1356\") "
Oct 30 21:30:57.695744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.695716    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb6aed9-10f1-4a0d-a95a-b4adb46a1356-kube-api-access-srhpc" pod "9cb6aed9-10f1-4a0d-a95a-b4adb46a1356" (UID: "9cb6aed9-10f1-4a0d-a95a-b4adb46a1356"). InnerVolumeSpecName "kube-api-access-srhpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:30:57.796466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.795932    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srhpc\" (UniqueName: \"kubernetes.io/projected/9cb6aed9-10f1-4a0d-a95a-b4adb46a1356-kube-api-access-srhpc\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:30:57.824532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:57.822500    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:58.274511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:58.274474    4918 generic.go:356] "Generic (PLEG): container finished" podID="9cb6aed9-10f1-4a0d-a95a-b4adb46a1356" containerID="2dafb4b6057861248e5e97047c86f5b5686f952359bca5f0003553936a476823" exitCode=137
Oct 30 21:30:58.274979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:58.274515    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-790016"
Oct 30 21:30:58.274979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:58.274572    4918 scope.go:122] "RemoveContainer" containerID="2dafb4b6057861248e5e97047c86f5b5686f952359bca5f0003553936a476823"
Oct 30 21:30:58.285646 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:58.285620    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-790016\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="9cb6aed9-10f1-4a0d-a95a-b4adb46a1356" pod="opendatahub/test-curl-7660-790016"
Oct 30 21:30:58.286116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:58.285966    4918 scope.go:122] "RemoveContainer" containerID="2dafb4b6057861248e5e97047c86f5b5686f952359bca5f0003553936a476823"
Oct 30 21:30:58.286295 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:58.286275    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dafb4b6057861248e5e97047c86f5b5686f952359bca5f0003553936a476823\": container with ID starting with 2dafb4b6057861248e5e97047c86f5b5686f952359bca5f0003553936a476823 not found: ID does not exist" containerID="2dafb4b6057861248e5e97047c86f5b5686f952359bca5f0003553936a476823"
Oct 30 21:30:58.286351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:58.286302    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dafb4b6057861248e5e97047c86f5b5686f952359bca5f0003553936a476823"} err="failed to get container status \"2dafb4b6057861248e5e97047c86f5b5686f952359bca5f0003553936a476823\": rpc error: code = NotFound desc = could not find container \"2dafb4b6057861248e5e97047c86f5b5686f952359bca5f0003553936a476823\": container with ID starting with 2dafb4b6057861248e5e97047c86f5b5686f952359bca5f0003553936a476823 not found: ID does not exist"
Oct 30 21:30:58.825996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:58.825968    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:58.828625 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:58.828590    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7660-790016\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="9cb6aed9-10f1-4a0d-a95a-b4adb46a1356" pod="opendatahub/test-curl-7660-790016"
Oct 30 21:30:59.081326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:59.081248    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c4ead3c-02e4-48a7-8a50-5c090b997489" path="/var/lib/kubelet/pods/5c4ead3c-02e4-48a7-8a50-5c090b997489/volumes"
Oct 30 21:30:59.081698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:59.081684    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb6aed9-10f1-4a0d-a95a-b4adb46a1356" path="/var/lib/kubelet/pods/9cb6aed9-10f1-4a0d-a95a-b4adb46a1356/volumes"
Oct 30 21:30:59.162306 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:30:59.162271    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:30:59.284838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:59.284807    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:30:59.285723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:30:59.285706    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:31:01.026140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.026096    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"]
Oct 30 21:31:01.035193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.035159    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:31:01.037948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.037921    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"]
Oct 30 21:31:01.039221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.039199    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e596df\""
Oct 30 21:31:01.040775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.040752    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e596df-dockercfg-9qsrl\""
Oct 30 21:31:01.066203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.066176    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"]
Oct 30 21:31:01.075696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.075668    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"]
Oct 30 21:31:01.075983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.075793    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"
Oct 30 21:31:01.168423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.165973    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2r9r\" (UniqueName: \"kubernetes.io/projected/20c16915-4492-4f87-bfca-b2f8952b0cc3-kube-api-access-l2r9r\") pod \"payload-processing-e2e-extmod-a-e596df-559989b645-xb22w\" (UID: \"20c16915-4492-4f87-bfca-b2f8952b0cc3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"
Oct 30 21:31:01.168423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.166282    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c16915-4492-4f87-bfca-b2f8952b0cc3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e596df-559989b645-xb22w\" (UID: \"20c16915-4492-4f87-bfca-b2f8952b0cc3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"
Oct 30 21:31:01.168423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.166330    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vjv\" (UniqueName: \"kubernetes.io/projected/aea69cff-5ca5-49b5-b038-bf4598440e71-kube-api-access-96vjv\") pod \"payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d\" (UID: \"aea69cff-5ca5-49b5-b038-bf4598440e71\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:31:01.168423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.166379    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea69cff-5ca5-49b5-b038-bf4598440e71-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d\" (UID: \"aea69cff-5ca5-49b5-b038-bf4598440e71\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:31:01.271416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.269654    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2r9r\" (UniqueName: \"kubernetes.io/projected/20c16915-4492-4f87-bfca-b2f8952b0cc3-kube-api-access-l2r9r\") pod \"payload-processing-e2e-extmod-a-e596df-559989b645-xb22w\" (UID: \"20c16915-4492-4f87-bfca-b2f8952b0cc3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"
Oct 30 21:31:01.271416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.269973    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c16915-4492-4f87-bfca-b2f8952b0cc3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e596df-559989b645-xb22w\" (UID: \"20c16915-4492-4f87-bfca-b2f8952b0cc3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"
Oct 30 21:31:01.271416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.270015    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vjv\" (UniqueName: \"kubernetes.io/projected/aea69cff-5ca5-49b5-b038-bf4598440e71-kube-api-access-96vjv\") pod \"payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d\" (UID: \"aea69cff-5ca5-49b5-b038-bf4598440e71\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:31:01.271416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.270062    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea69cff-5ca5-49b5-b038-bf4598440e71-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d\" (UID: \"aea69cff-5ca5-49b5-b038-bf4598440e71\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:31:01.271416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.271056    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea69cff-5ca5-49b5-b038-bf4598440e71-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d\" (UID: \"aea69cff-5ca5-49b5-b038-bf4598440e71\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:31:01.297464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.279921    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c16915-4492-4f87-bfca-b2f8952b0cc3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e596df-559989b645-xb22w\" (UID: \"20c16915-4492-4f87-bfca-b2f8952b0cc3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"
Oct 30 21:31:01.366412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.348410    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vjv\" (UniqueName: \"kubernetes.io/projected/aea69cff-5ca5-49b5-b038-bf4598440e71-kube-api-access-96vjv\") pod \"payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d\" (UID: \"aea69cff-5ca5-49b5-b038-bf4598440e71\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:31:01.367052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.367025    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:31:01.387412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.385151    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2r9r\" (UniqueName: \"kubernetes.io/projected/20c16915-4492-4f87-bfca-b2f8952b0cc3-kube-api-access-l2r9r\") pod \"payload-processing-e2e-extmod-a-e596df-559989b645-xb22w\" (UID: \"20c16915-4492-4f87-bfca-b2f8952b0cc3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"
Oct 30 21:31:01.415025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.409467    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"
Oct 30 21:31:01.800032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.800004    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"]
Oct 30 21:31:01.800427 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:31:01.800028    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c16915_4492_4f87_bfca_b2f8952b0cc3.slice/crio-80b98e5988788dbca21f9b7a4641dbd2ae96a5aac9837bfee3942cbaebf5d7d9 WatchSource:0}: Error finding container 80b98e5988788dbca21f9b7a4641dbd2ae96a5aac9837bfee3942cbaebf5d7d9: Status 404 returned error can't find the container with id 80b98e5988788dbca21f9b7a4641dbd2ae96a5aac9837bfee3942cbaebf5d7d9
Oct 30 21:31:01.806533 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:31:01.806503    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea69cff_5ca5_49b5_b038_bf4598440e71.slice/crio-7e654b3c964d74afeb37b2a4f0a18eacf542f512553939b01711b932ace8ad02 WatchSource:0}: Error finding container 7e654b3c964d74afeb37b2a4f0a18eacf542f512553939b01711b932ace8ad02: Status 404 returned error can't find the container with id 7e654b3c964d74afeb37b2a4f0a18eacf542f512553939b01711b932ace8ad02
Oct 30 21:31:01.807030 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:01.807000    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"]
Oct 30 21:31:02.370630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:02.370579    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w" event={"ID":"20c16915-4492-4f87-bfca-b2f8952b0cc3","Type":"ContainerStarted","Data":"d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa"}
Oct 30 21:31:02.371093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:02.370650    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"
Oct 30 21:31:02.371093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:02.370675    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w" event={"ID":"20c16915-4492-4f87-bfca-b2f8952b0cc3","Type":"ContainerStarted","Data":"80b98e5988788dbca21f9b7a4641dbd2ae96a5aac9837bfee3942cbaebf5d7d9"}
Oct 30 21:31:02.372241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:02.372218    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d" event={"ID":"aea69cff-5ca5-49b5-b038-bf4598440e71","Type":"ContainerStarted","Data":"098d4189b90c7ed6a98d61c01c7087813f6228cb7488661aabc37a8404c57ea8"}
Oct 30 21:31:02.372354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:02.372267    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:31:02.372354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:02.372289    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d" event={"ID":"aea69cff-5ca5-49b5-b038-bf4598440e71","Type":"ContainerStarted","Data":"7e654b3c964d74afeb37b2a4f0a18eacf542f512553939b01711b932ace8ad02"}
Oct 30 21:31:02.393171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:02.393123    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w" podStartSLOduration=1.393111156 podStartE2EDuration="1.393111156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:02.387754909 +0000 UTC m=+1775.434314055" watchObservedRunningTime="2026-10-30 21:31:02.393111156 +0000 UTC m=+1775.439670350"
Oct 30 21:31:02.415094 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:02.415049    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d" podStartSLOduration=1.415035797 podStartE2EDuration="1.415035797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:02.41060767 +0000 UTC m=+1775.457166827" watchObservedRunningTime="2026-10-30 21:31:02.415035797 +0000 UTC m=+1775.461594941"
Oct 30 21:31:08.024735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:08.024695    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"]
Oct 30 21:31:08.028760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:08.028735    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6152c89d-tzscz"]
Oct 30 21:31:09.124356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:09.124318    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d66548e1-39b4-40d8-a84b-c91df71a5726" path="/var/lib/kubelet/pods/d66548e1-39b4-40d8-a84b-c91df71a5726/volumes"
Oct 30 21:31:12.066224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.065434    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"]
Oct 30 21:31:12.071559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.071419    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f84cce-9t7jr"]
Oct 30 21:31:12.653225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.653193    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"]
Oct 30 21:31:12.659532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.659500    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:12.665192 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.665168    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:12.665355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.665277    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:31:12.665355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.665341    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:31:12.666193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.666172    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:31:12.666618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.666593    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"]
Oct 30 21:31:12.747801 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.747765    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9529bf3e-kknjm\" (UID: \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:12.748051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.748033    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9529bf3e-kknjm\" (UID: \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:12.748112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.748097    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb468\" (UniqueName: \"kubernetes.io/projected/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-kube-api-access-jb468\") pod \"maas-api-revoke-sub-9529bf3e-kknjm\" (UID: \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:12.853413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.849978    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9529bf3e-kknjm\" (UID: \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:12.853413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.850293    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9529bf3e-kknjm\" (UID: \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:12.853413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.850414    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb468\" (UniqueName: \"kubernetes.io/projected/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-kube-api-access-jb468\") pod \"maas-api-revoke-sub-9529bf3e-kknjm\" (UID: \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:12.853413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.851333    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9529bf3e-kknjm\" (UID: \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:12.865280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.865080    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9529bf3e-kknjm\" (UID: \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:12.885688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.877360    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb468\" (UniqueName: \"kubernetes.io/projected/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-kube-api-access-jb468\") pod \"maas-api-revoke-sub-9529bf3e-kknjm\" (UID: \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:12.982929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:12.982902    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:13.034302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:13.034269    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"]
Oct 30 21:31:13.038734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:13.038711    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f3b5a81-pcmwg"]
Oct 30 21:31:13.086717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:13.086434    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85370ff2-fa9e-43e0-815e-3add8f2fc416" path="/var/lib/kubelet/pods/85370ff2-fa9e-43e0-815e-3add8f2fc416/volumes"
Oct 30 21:31:13.087080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:13.086958    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e194051c-28aa-4275-8a63-ce4ce433e23d" path="/var/lib/kubelet/pods/e194051c-28aa-4275-8a63-ce4ce433e23d/volumes"
Oct 30 21:31:13.271524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:13.271451    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"]
Oct 30 21:31:13.275709 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:31:13.275679    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf613b20d_5ebd_4dcb_9a52_b08ee1e61e7d.slice/crio-7d18e2540661d94af2862ec40db00466b96ef97b8c068ca313c0dd189ccde5b1 WatchSource:0}: Error finding container 7d18e2540661d94af2862ec40db00466b96ef97b8c068ca313c0dd189ccde5b1: Status 404 returned error can't find the container with id 7d18e2540661d94af2862ec40db00466b96ef97b8c068ca313c0dd189ccde5b1
Oct 30 21:31:13.413132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:13.413105    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:31:13.413305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:13.413166    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"
Oct 30 21:31:13.691578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:13.691549    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm" event={"ID":"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d","Type":"ContainerStarted","Data":"7d18e2540661d94af2862ec40db00466b96ef97b8c068ca313c0dd189ccde5b1"}
Oct 30 21:31:14.392177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.392146    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"]
Oct 30 21:31:14.396834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.396818    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:14.406899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.406857    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"]
Oct 30 21:31:14.519638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.519604    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld6k7\" (UniqueName: \"kubernetes.io/projected/dfc9f470-5c19-40fc-a15a-dcc783628049-kube-api-access-ld6k7\") pod \"maas-api-revoke-sub-83668192-ck8fs\" (UID: \"dfc9f470-5c19-40fc-a15a-dcc783628049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:14.519638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.519640    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83668192-ck8fs\" (UID: \"dfc9f470-5c19-40fc-a15a-dcc783628049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:14.519938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.519915    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83668192-ck8fs\" (UID: \"dfc9f470-5c19-40fc-a15a-dcc783628049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:14.624935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.624831    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83668192-ck8fs\" (UID: \"dfc9f470-5c19-40fc-a15a-dcc783628049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:14.625483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.625455    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6k7\" (UniqueName: \"kubernetes.io/projected/dfc9f470-5c19-40fc-a15a-dcc783628049-kube-api-access-ld6k7\") pod \"maas-api-revoke-sub-83668192-ck8fs\" (UID: \"dfc9f470-5c19-40fc-a15a-dcc783628049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:14.625650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.625635    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83668192-ck8fs\" (UID: \"dfc9f470-5c19-40fc-a15a-dcc783628049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:14.634419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.626757    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83668192-ck8fs\" (UID: \"dfc9f470-5c19-40fc-a15a-dcc783628049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:14.655249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.655133    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld6k7\" (UniqueName: \"kubernetes.io/projected/dfc9f470-5c19-40fc-a15a-dcc783628049-kube-api-access-ld6k7\") pod \"maas-api-revoke-sub-83668192-ck8fs\" (UID: \"dfc9f470-5c19-40fc-a15a-dcc783628049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:14.682740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.682703    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83668192-ck8fs\" (UID: \"dfc9f470-5c19-40fc-a15a-dcc783628049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:14.707235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.707203    4918 generic.go:356] "Generic (PLEG): container finished" podID="f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d" containerID="bed3d8852d52e02b6ab3b5724e4e335bdfa07510b6dcb3d3b87b6d3d57b99cec" exitCode=0
Oct 30 21:31:14.707457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.707239    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm" event={"ID":"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d","Type":"ContainerDied","Data":"bed3d8852d52e02b6ab3b5724e4e335bdfa07510b6dcb3d3b87b6d3d57b99cec"}
Oct 30 21:31:14.766048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:14.766012    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:15.108574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:15.108550    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"]
Oct 30 21:31:15.717383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:15.717253    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs" event={"ID":"dfc9f470-5c19-40fc-a15a-dcc783628049","Type":"ContainerStarted","Data":"3fe84d63fe3208b53c91650ce5628e2055321ad460ddcbe13eb614365fefa840"}
Oct 30 21:31:15.717383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:15.717332    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs" event={"ID":"dfc9f470-5c19-40fc-a15a-dcc783628049","Type":"ContainerStarted","Data":"5f330d297e7343bd54f4ff71142e721d540d3b57436942d93b6a87b7a0cd0305"}
Oct 30 21:31:15.739228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:15.739181    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs" podStartSLOduration=1.739167268 podStartE2EDuration="1.739167268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:15.737213564 +0000 UTC m=+1788.783772721" watchObservedRunningTime="2026-10-30 21:31:15.739167268 +0000 UTC m=+1788.785726412"
Oct 30 21:31:15.965580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:15.965554    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:16.070543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.070509    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-openshift-ca\") pod \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\" (UID: \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\") "
Oct 30 21:31:16.070740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.070647    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-kube-api-access-jb468\" (UniqueName: \"kubernetes.io/projected/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-kube-api-access-jb468\") pod \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\" (UID: \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\") "
Oct 30 21:31:16.070740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.070692    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-xks-ca\") pod \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\" (UID: \"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d\") "
Oct 30 21:31:16.070958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.070916    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-openshift-ca" pod "f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d" (UID: "f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:16.071504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.071471    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:16.073072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.073047    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-kube-api-access-jb468" pod "f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d" (UID: "f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d"). InnerVolumeSpecName "kube-api-access-jb468". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:16.073177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.073107    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-xks-ca" pod "f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d" (UID: "f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:16.173191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.173159    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb468\" (UniqueName: \"kubernetes.io/projected/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-kube-api-access-jb468\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:16.173191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.173188    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:16.727883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.727854    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"
Oct 30 21:31:16.728346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.727897    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm" event={"ID":"f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d","Type":"ContainerDied","Data":"7d18e2540661d94af2862ec40db00466b96ef97b8c068ca313c0dd189ccde5b1"}
Oct 30 21:31:16.728346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.727951    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d18e2540661d94af2862ec40db00466b96ef97b8c068ca313c0dd189ccde5b1"
Oct 30 21:31:16.730044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.730009    4918 generic.go:356] "Generic (PLEG): container finished" podID="dfc9f470-5c19-40fc-a15a-dcc783628049" containerID="3fe84d63fe3208b53c91650ce5628e2055321ad460ddcbe13eb614365fefa840" exitCode=0
Oct 30 21:31:16.730179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:16.730096    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs" event={"ID":"dfc9f470-5c19-40fc-a15a-dcc783628049","Type":"ContainerDied","Data":"3fe84d63fe3208b53c91650ce5628e2055321ad460ddcbe13eb614365fefa840"}
Oct 30 21:31:17.931230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:17.931205    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:18.111938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:18.111861    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-openshift-ca\") pod \"dfc9f470-5c19-40fc-a15a-dcc783628049\" (UID: \"dfc9f470-5c19-40fc-a15a-dcc783628049\") "
Oct 30 21:31:18.112112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:18.112077    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-xks-ca\") pod \"dfc9f470-5c19-40fc-a15a-dcc783628049\" (UID: \"dfc9f470-5c19-40fc-a15a-dcc783628049\") "
Oct 30 21:31:18.112180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:18.112121    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc9f470-5c19-40fc-a15a-dcc783628049-kube-api-access-ld6k7\" (UniqueName: \"kubernetes.io/projected/dfc9f470-5c19-40fc-a15a-dcc783628049-kube-api-access-ld6k7\") pod \"dfc9f470-5c19-40fc-a15a-dcc783628049\" (UID: \"dfc9f470-5c19-40fc-a15a-dcc783628049\") "
Oct 30 21:31:18.112254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:18.112226    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-openshift-ca" pod "dfc9f470-5c19-40fc-a15a-dcc783628049" (UID: "dfc9f470-5c19-40fc-a15a-dcc783628049"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:18.112598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:18.112578    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:18.114442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:18.114420    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-xks-ca" pod "dfc9f470-5c19-40fc-a15a-dcc783628049" (UID: "dfc9f470-5c19-40fc-a15a-dcc783628049"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:18.114631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:18.114607    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc9f470-5c19-40fc-a15a-dcc783628049-kube-api-access-ld6k7" pod "dfc9f470-5c19-40fc-a15a-dcc783628049" (UID: "dfc9f470-5c19-40fc-a15a-dcc783628049"). InnerVolumeSpecName "kube-api-access-ld6k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:18.234721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:18.234545    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc9f470-5c19-40fc-a15a-dcc783628049-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:18.234721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:18.234604    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld6k7\" (UniqueName: \"kubernetes.io/projected/dfc9f470-5c19-40fc-a15a-dcc783628049-kube-api-access-ld6k7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:18.748482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:18.748455    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"
Oct 30 21:31:18.748482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:18.748474    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs" event={"ID":"dfc9f470-5c19-40fc-a15a-dcc783628049","Type":"ContainerDied","Data":"5f330d297e7343bd54f4ff71142e721d540d3b57436942d93b6a87b7a0cd0305"}
Oct 30 21:31:18.748705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:18.748523    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f330d297e7343bd54f4ff71142e721d540d3b57436942d93b6a87b7a0cd0305"
Oct 30 21:31:20.097058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:20.097019    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"]
Oct 30 21:31:20.103553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:20.103521    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9661254-b9fhh"]
Oct 30 21:31:21.144545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:21.144511    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b9e0be-eef3-4fa2-902b-99364944dec9" path="/var/lib/kubelet/pods/33b9e0be-eef3-4fa2-902b-99364944dec9/volumes"
Oct 30 21:31:22.961715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.961685    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"]
Oct 30 21:31:22.962480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.962443    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d" containerName="revoke-keys"
Oct 30 21:31:22.962480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.962459    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d" containerName="revoke-keys"
Oct 30 21:31:22.962480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.962480    4918 cpu_manager.go:414] "containerMap: removing container" podUID="dfc9f470-5c19-40fc-a15a-dcc783628049" containerName="revoke-keys"
Oct 30 21:31:22.962661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.962486    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc9f470-5c19-40fc-a15a-dcc783628049" containerName="revoke-keys"
Oct 30 21:31:22.962661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.962626    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d" containerName="revoke-keys"
Oct 30 21:31:22.962661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.962636    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc9f470-5c19-40fc-a15a-dcc783628049" containerName="revoke-keys"
Oct 30 21:31:22.966836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.966817    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:22.972603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.972406    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:22.972603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.972444    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:31:22.972603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.972447    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:31:22.972603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.972576    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:31:22.975821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:22.975800    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"]
Oct 30 21:31:23.027331 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.027300    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"]
Oct 30 21:31:23.031468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.031440    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f59e43a-27gk7"]
Oct 30 21:31:23.032963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.032939    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c06b8c-229h5\" (UID: \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:23.033152 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.033127    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c06b8c-229h5\" (UID: \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:23.033351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.033336    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whgwt\" (UniqueName: \"kubernetes.io/projected/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-kube-api-access-whgwt\") pod \"maas-api-revoke-sub-71c06b8c-229h5\" (UID: \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:23.080963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.080936    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aed7b57-d027-49d4-92c0-7848f4cb3c78" path="/var/lib/kubelet/pods/1aed7b57-d027-49d4-92c0-7848f4cb3c78/volumes"
Oct 30 21:31:23.134981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.134948    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c06b8c-229h5\" (UID: \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:23.135184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.135159    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c06b8c-229h5\" (UID: \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:23.135346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.135327    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whgwt\" (UniqueName: \"kubernetes.io/projected/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-kube-api-access-whgwt\") pod \"maas-api-revoke-sub-71c06b8c-229h5\" (UID: \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:23.135730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.135707    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71c06b8c-229h5\" (UID: \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:23.137814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.137784    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71c06b8c-229h5\" (UID: \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:23.144156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.144123    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whgwt\" (UniqueName: \"kubernetes.io/projected/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-kube-api-access-whgwt\") pod \"maas-api-revoke-sub-71c06b8c-229h5\" (UID: \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:23.285630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.285599    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:23.586520 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:31:23.586493    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665e9bcb_8c55_4ba9_bdc1_7dc9d4a3a709.slice/crio-a1fbdbde7c4d4c84530347cfc8207ce2f7f8893b2c5e203a2e0db8924bfc892e WatchSource:0}: Error finding container a1fbdbde7c4d4c84530347cfc8207ce2f7f8893b2c5e203a2e0db8924bfc892e: Status 404 returned error can't find the container with id a1fbdbde7c4d4c84530347cfc8207ce2f7f8893b2c5e203a2e0db8924bfc892e
Oct 30 21:31:23.586682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.586659    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"]
Oct 30 21:31:23.802106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:23.802075    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5" event={"ID":"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709","Type":"ContainerStarted","Data":"a1fbdbde7c4d4c84530347cfc8207ce2f7f8893b2c5e203a2e0db8924bfc892e"}
Oct 30 21:31:24.691616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.691589    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"]
Oct 30 21:31:24.700265 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.700234    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:24.706934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.706901    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"]
Oct 30 21:31:24.763501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.763417    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a24e1e6-5tfm9\" (UID: \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:24.763662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.763540    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq6p7\" (UniqueName: \"kubernetes.io/projected/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-kube-api-access-qq6p7\") pod \"maas-api-revoke-sub-2a24e1e6-5tfm9\" (UID: \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:24.763662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.763604    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a24e1e6-5tfm9\" (UID: \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:24.812188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.812153    4918 generic.go:356] "Generic (PLEG): container finished" podID="665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709" containerID="4b6eb0018c3695b426b6025e0a9ce57bc9ca688b66be804a1d638e617248f75e" exitCode=0
Oct 30 21:31:24.812375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.812242    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5" event={"ID":"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709","Type":"ContainerDied","Data":"4b6eb0018c3695b426b6025e0a9ce57bc9ca688b66be804a1d638e617248f75e"}
Oct 30 21:31:24.893416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.885032    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a24e1e6-5tfm9\" (UID: \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:24.917413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.885136    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq6p7\" (UniqueName: \"kubernetes.io/projected/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-kube-api-access-qq6p7\") pod \"maas-api-revoke-sub-2a24e1e6-5tfm9\" (UID: \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:24.917413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.912581    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a24e1e6-5tfm9\" (UID: \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:24.917413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.913690    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a24e1e6-5tfm9\" (UID: \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:24.950824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.950788    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq6p7\" (UniqueName: \"kubernetes.io/projected/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-kube-api-access-qq6p7\") pod \"maas-api-revoke-sub-2a24e1e6-5tfm9\" (UID: \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:25.003418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:24.992938    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a24e1e6-5tfm9\" (UID: \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:25.027530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:25.023694    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:25.591888 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:31:25.591853    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1dbc5a_6e0e_4647_93c2_cce9a11cc705.slice/crio-fca04ffb4d9a477434ac37ad4e40c86d907807f63fda86944c58f39efc63b8ea WatchSource:0}: Error finding container fca04ffb4d9a477434ac37ad4e40c86d907807f63fda86944c58f39efc63b8ea: Status 404 returned error can't find the container with id fca04ffb4d9a477434ac37ad4e40c86d907807f63fda86944c58f39efc63b8ea
Oct 30 21:31:25.592223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:25.592204    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"]
Oct 30 21:31:25.824298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:25.824266    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9" event={"ID":"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705","Type":"ContainerStarted","Data":"fca04ffb4d9a477434ac37ad4e40c86d907807f63fda86944c58f39efc63b8ea"}
Oct 30 21:31:25.981316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:25.981291    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:26.058738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.058711    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-kube-api-access-whgwt\" (UniqueName: \"kubernetes.io/projected/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-kube-api-access-whgwt\") pod \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\" (UID: \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\") "
Oct 30 21:31:26.058931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.058754    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-openshift-ca\") pod \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\" (UID: \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\") "
Oct 30 21:31:26.058931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.058792    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-xks-ca\") pod \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\" (UID: \"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709\") "
Oct 30 21:31:26.059362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.059332    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-openshift-ca" pod "665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709" (UID: "665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:26.059864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.059844    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:26.061853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.061826    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-xks-ca" pod "665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709" (UID: "665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:26.097538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.083610    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-kube-api-access-whgwt" pod "665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709" (UID: "665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709"). InnerVolumeSpecName "kube-api-access-whgwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:26.162283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.162248    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whgwt\" (UniqueName: \"kubernetes.io/projected/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-kube-api-access-whgwt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:26.162549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.162530    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:26.848867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.848782    4918 generic.go:356] "Generic (PLEG): container finished" podID="4b1dbc5a-6e0e-4647-93c2-cce9a11cc705" containerID="9f5c5e7bc6482fff55784606cf9fad8ddc9c94389a4965604bdabfe4d58bea25" exitCode=0
Oct 30 21:31:26.849355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.848938    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9" event={"ID":"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705","Type":"ContainerDied","Data":"9f5c5e7bc6482fff55784606cf9fad8ddc9c94389a4965604bdabfe4d58bea25"}
Oct 30 21:31:26.851182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.851159    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"
Oct 30 21:31:26.851329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.851216    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5" event={"ID":"665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709","Type":"ContainerDied","Data":"a1fbdbde7c4d4c84530347cfc8207ce2f7f8893b2c5e203a2e0db8924bfc892e"}
Oct 30 21:31:26.851329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:26.851298    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1fbdbde7c4d4c84530347cfc8207ce2f7f8893b2c5e203a2e0db8924bfc892e"
Oct 30 21:31:27.034859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:27.034815    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"]
Oct 30 21:31:27.047445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:27.047416    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af46d705-q4hff"]
Oct 30 21:31:27.080172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:27.080140    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd9d723-0315-4c6c-8c77-a82370aecac4" path="/var/lib/kubelet/pods/ebd9d723-0315-4c6c-8c77-a82370aecac4/volumes"
Oct 30 21:31:28.032402 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.028893    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"]
Oct 30 21:31:28.041105 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.041079    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10a20d3-74fb2"]
Oct 30 21:31:28.134614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.123861    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:28.218167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.218129    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-xks-ca\") pod \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\" (UID: \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\") "
Oct 30 21:31:28.218428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.218224    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-kube-api-access-qq6p7\" (UniqueName: \"kubernetes.io/projected/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-kube-api-access-qq6p7\") pod \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\" (UID: \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\") "
Oct 30 21:31:28.218428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.218304    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-openshift-ca\") pod \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\" (UID: \"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705\") "
Oct 30 21:31:28.219098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.219069    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-openshift-ca" pod "4b1dbc5a-6e0e-4647-93c2-cce9a11cc705" (UID: "4b1dbc5a-6e0e-4647-93c2-cce9a11cc705"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:28.219352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.219323    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:28.221514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.221485    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-xks-ca" pod "4b1dbc5a-6e0e-4647-93c2-cce9a11cc705" (UID: "4b1dbc5a-6e0e-4647-93c2-cce9a11cc705"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:28.221623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.221491    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-kube-api-access-qq6p7" pod "4b1dbc5a-6e0e-4647-93c2-cce9a11cc705" (UID: "4b1dbc5a-6e0e-4647-93c2-cce9a11cc705"). InnerVolumeSpecName "kube-api-access-qq6p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:28.320874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.320841    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq6p7\" (UniqueName: \"kubernetes.io/projected/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-kube-api-access-qq6p7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:28.320874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.320868    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:28.870920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.870894    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"
Oct 30 21:31:28.871107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.870927    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9" event={"ID":"4b1dbc5a-6e0e-4647-93c2-cce9a11cc705","Type":"ContainerDied","Data":"fca04ffb4d9a477434ac37ad4e40c86d907807f63fda86944c58f39efc63b8ea"}
Oct 30 21:31:28.871107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:28.870977    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fca04ffb4d9a477434ac37ad4e40c86d907807f63fda86944c58f39efc63b8ea"
Oct 30 21:31:29.079489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:29.079444    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9170164-5c52-4eee-879d-a88656e9e4f8" path="/var/lib/kubelet/pods/c9170164-5c52-4eee-879d-a88656e9e4f8/volumes"
Oct 30 21:31:29.163596 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:31:29.163514    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:31:30.180631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:30.180598    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:31:30.181046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:30.180941    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:31:31.007888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.007855    4918 scope.go:122] "RemoveContainer" containerID="636d110aaa712187fc5c6b4f8bab87095f7b928e5a1fcc822dedcea2d3ffa94b"
Oct 30 21:31:31.019436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.019410    4918 scope.go:122] "RemoveContainer" containerID="c6c7a9aba95675f177fd888f3456665b7321b891973e34ff6fabfa388f936edb"
Oct 30 21:31:31.099650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.097111    4918 scope.go:122] "RemoveContainer" containerID="ed360326e11369a7f3d74df34f525305ccf4036ae9a81edd3d248064dc9de8c5"
Oct 30 21:31:31.129900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.129880    4918 scope.go:122] "RemoveContainer" containerID="5a3bd51aa6adbd82acc874b0184c42f45c31624e87405dff80376ce3473f9b08"
Oct 30 21:31:31.139847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.139829    4918 scope.go:122] "RemoveContainer" containerID="6044b40e4a15c6750d3370d90be2a2ffd557155fa03a4b32dc01d9f53bb21192"
Oct 30 21:31:31.149742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.149715    4918 scope.go:122] "RemoveContainer" containerID="41a0ac9180dd107b9a6d6a3dfa6058b8180055e3ec3551281d5eab0504e13936"
Oct 30 21:31:31.159549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.159529    4918 scope.go:122] "RemoveContainer" containerID="b877e8339a424ad92014e16357645e10492e47b52e3aae37c8e430c8cbd4fdab"
Oct 30 21:31:31.171892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.171873    4918 scope.go:122] "RemoveContainer" containerID="ea08d7260f3293005d811b73031e834614bcbb42eb9665260f6d169f76ae1667"
Oct 30 21:31:31.181910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.181890    4918 scope.go:122] "RemoveContainer" containerID="db10839681c9e8806555311d58bfb790454ea5d5a297aa4902d37f9aa5d5f019"
Oct 30 21:31:31.856336 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.856294    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"]
Oct 30 21:31:31.857562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.857538    4918 cpu_manager.go:414] "containerMap: removing container" podUID="665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709" containerName="revoke-keys"
Oct 30 21:31:31.857660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.857564    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709" containerName="revoke-keys"
Oct 30 21:31:31.857660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.857633    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4b1dbc5a-6e0e-4647-93c2-cce9a11cc705" containerName="revoke-keys"
Oct 30 21:31:31.857660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.857641    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1dbc5a-6e0e-4647-93c2-cce9a11cc705" containerName="revoke-keys"
Oct 30 21:31:31.857850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.857835    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1dbc5a-6e0e-4647-93c2-cce9a11cc705" containerName="revoke-keys"
Oct 30 21:31:31.857895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.857866    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709" containerName="revoke-keys"
Oct 30 21:31:31.864517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.864489    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:31.876429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.876366    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"]
Oct 30 21:31:31.882702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.882676    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f1ed97-openshift-default-dockercfg-h5n84\""
Oct 30 21:31:31.897100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.897054    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-data\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:31.897258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.897104    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-certs\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:31.897258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.897238    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-credential-socket\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:31.897467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.897446    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-token\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:31.897582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.897563    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-socket\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:31.897649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.897624    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sx76\" (UniqueName: \"kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-kube-api-access-5sx76\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:31.897761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.897744    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-podinfo\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:31.897842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.897778    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d7835c-782d-400b-8c1e-3a4654fd8a12-istiod-ca-cert\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:31.897903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:31.897878    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-envoy\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.104417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.101686    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-data\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.104417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.101756    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-certs\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.104417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.101821    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-credential-socket\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.104417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.102067    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-token\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.104417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.102148    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-socket\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.104417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.102206    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sx76\" (UniqueName: \"kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-kube-api-access-5sx76\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.104417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.102494    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-podinfo\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.104417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.102533    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d7835c-782d-400b-8c1e-3a4654fd8a12-istiod-ca-cert\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.104417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.102619    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-envoy\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.131552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.113467    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-data\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.131552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.113900    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-certs\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.131552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.114335    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-credential-socket\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.151448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.137932    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-socket\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.151448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.144540    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d7835c-782d-400b-8c1e-3a4654fd8a12-istiod-ca-cert\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.258199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.258156    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-token\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.300008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.292318    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sx76\" (UniqueName: \"kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-kube-api-access-5sx76\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.300771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.300736    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-envoy\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.329414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.319998    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-podinfo\") pod \"e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") " pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.488570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.488535    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:32.874062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.874032    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"]
Oct 30 21:31:32.876463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.876421    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:31:32.876606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.876512    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:31:32.876606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.876553    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:31:32.922362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:32.922325    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" event={"ID":"28d7835c-782d-400b-8c1e-3a4654fd8a12","Type":"ContainerStarted","Data":"d7118edf83c51d91e6effe8db4e66b1be9018c9495662186028bc46dd221bdbf"}
Oct 30 21:31:33.934783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:33.934742    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" event={"ID":"28d7835c-782d-400b-8c1e-3a4654fd8a12","Type":"ContainerStarted","Data":"fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317"}
Oct 30 21:31:33.960249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:33.960204    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" podStartSLOduration=2.960188933 podStartE2EDuration="2.960188933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:33.956321309 +0000 UTC m=+1807.002880484" watchObservedRunningTime="2026-10-30 21:31:33.960188933 +0000 UTC m=+1807.006748078"
Oct 30 21:31:34.488850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:34.488821    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:34.494596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:34.494574    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:35.004946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:35.003309    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:35.017446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:35.011672    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:31:37.585615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.585580    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"]
Oct 30 21:31:37.590149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.590131    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"
Oct 30 21:31:37.600049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.600019    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-f1ed97-dockercfg-mcdnv\""
Oct 30 21:31:37.600183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.600087    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-f1ed97\""
Oct 30 21:31:37.605824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.605792    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"]
Oct 30 21:31:37.610534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.610510    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"]
Oct 30 21:31:37.618914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.618888    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"
Oct 30 21:31:37.636295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.636266    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"]
Oct 30 21:31:37.784499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.764726    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc\" (UID: \"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"
Oct 30 21:31:37.784499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.764907    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb7ng\" (UniqueName: \"kubernetes.io/projected/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-kube-api-access-lb7ng\") pod \"payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc\" (UID: \"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"
Oct 30 21:31:37.784499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.764941    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724e33b5-e637-47fb-b63f-dab247b15cd5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr\" (UID: \"724e33b5-e637-47fb-b63f-dab247b15cd5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"
Oct 30 21:31:37.784499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.765086    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5db4h\" (UniqueName: \"kubernetes.io/projected/724e33b5-e637-47fb-b63f-dab247b15cd5-kube-api-access-5db4h\") pod \"payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr\" (UID: \"724e33b5-e637-47fb-b63f-dab247b15cd5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"
Oct 30 21:31:37.866841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.866625    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc\" (UID: \"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"
Oct 30 21:31:37.867365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.867339    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb7ng\" (UniqueName: \"kubernetes.io/projected/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-kube-api-access-lb7ng\") pod \"payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc\" (UID: \"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"
Oct 30 21:31:37.867650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.867628    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724e33b5-e637-47fb-b63f-dab247b15cd5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr\" (UID: \"724e33b5-e637-47fb-b63f-dab247b15cd5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"
Oct 30 21:31:37.867910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.867892    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5db4h\" (UniqueName: \"kubernetes.io/projected/724e33b5-e637-47fb-b63f-dab247b15cd5-kube-api-access-5db4h\") pod \"payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr\" (UID: \"724e33b5-e637-47fb-b63f-dab247b15cd5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"
Oct 30 21:31:37.869300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.869277    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc\" (UID: \"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"
Oct 30 21:31:37.878444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.878415    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724e33b5-e637-47fb-b63f-dab247b15cd5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr\" (UID: \"724e33b5-e637-47fb-b63f-dab247b15cd5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"
Oct 30 21:31:37.893111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.893084    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5db4h\" (UniqueName: \"kubernetes.io/projected/724e33b5-e637-47fb-b63f-dab247b15cd5-kube-api-access-5db4h\") pod \"payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr\" (UID: \"724e33b5-e637-47fb-b63f-dab247b15cd5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"
Oct 30 21:31:37.913035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.913000    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb7ng\" (UniqueName: \"kubernetes.io/projected/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-kube-api-access-lb7ng\") pod \"payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc\" (UID: \"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"
Oct 30 21:31:37.913659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:37.913633    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"
Oct 30 21:31:38.028720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:38.027382    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"
Oct 30 21:31:38.384634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:38.384558    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"]
Oct 30 21:31:38.384907 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:31:38.384879    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724e33b5_e637_47fb_b63f_dab247b15cd5.slice/crio-f8338e7c8540e018ca923b48b291b717bf24202ad9484160f204bce9620b27a4 WatchSource:0}: Error finding container f8338e7c8540e018ca923b48b291b717bf24202ad9484160f204bce9620b27a4: Status 404 returned error can't find the container with id f8338e7c8540e018ca923b48b291b717bf24202ad9484160f204bce9620b27a4
Oct 30 21:31:38.593024 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:31:38.592992    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b022d69_5d42_42b2_bb5e_bbb98c4cedb8.slice/crio-e9059fdd043ff641f60c1f16fe90b5238a95e88ec625515d2683f3e175c61f7e WatchSource:0}: Error finding container e9059fdd043ff641f60c1f16fe90b5238a95e88ec625515d2683f3e175c61f7e: Status 404 returned error can't find the container with id e9059fdd043ff641f60c1f16fe90b5238a95e88ec625515d2683f3e175c61f7e
Oct 30 21:31:38.593354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:38.593080    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"]
Oct 30 21:31:39.180139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:39.180101    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc" event={"ID":"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8","Type":"ContainerStarted","Data":"c0537919f97d123320641e8e998c93579c889a6a95438d483ccca70f141a603c"}
Oct 30 21:31:39.180360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:39.180166    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc" event={"ID":"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8","Type":"ContainerStarted","Data":"e9059fdd043ff641f60c1f16fe90b5238a95e88ec625515d2683f3e175c61f7e"}
Oct 30 21:31:39.180360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:39.180196    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"
Oct 30 21:31:39.181366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:39.181340    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr" event={"ID":"724e33b5-e637-47fb-b63f-dab247b15cd5","Type":"ContainerStarted","Data":"2dea06b0a1433e40f2324ed25e7b8cf1acc515ec6b65e4603cefaa53990bdaee"}
Oct 30 21:31:39.181554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:39.181385    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr" event={"ID":"724e33b5-e637-47fb-b63f-dab247b15cd5","Type":"ContainerStarted","Data":"f8338e7c8540e018ca923b48b291b717bf24202ad9484160f204bce9620b27a4"}
Oct 30 21:31:39.181554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:39.181463    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"
Oct 30 21:31:39.200558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:39.200046    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc" podStartSLOduration=2.200028437 podStartE2EDuration="2.200028437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:39.196886695 +0000 UTC m=+1812.243445868" watchObservedRunningTime="2026-10-30 21:31:39.200028437 +0000 UTC m=+1812.246587583"
Oct 30 21:31:39.217205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:39.217163    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr" podStartSLOduration=2.217149822 podStartE2EDuration="2.217149822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:39.214127005 +0000 UTC m=+1812.260686169" watchObservedRunningTime="2026-10-30 21:31:39.217149822 +0000 UTC m=+1812.263708967"
Oct 30 21:31:44.668486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.600966    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"]
Oct 30 21:31:44.806418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.797928    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:44.806418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.802865    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"]
Oct 30 21:31:44.829873 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.827788    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-729802-openshift-default-dockercfg-4wqzl\""
Oct 30 21:31:44.916417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.913280    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7574078-8dcf-4713-9574-c489ef73a943-istiod-ca-cert\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:44.916417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.913341    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-istio-token\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:44.916417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.913515    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-socket\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:44.916417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.913543    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-envoy\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:44.916417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.913674    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-data\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:44.916417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.913729    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-credential-socket\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:44.916417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.913838    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-certs\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:44.916417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.914161    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh56d\" (UniqueName: \"kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-kube-api-access-zh56d\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:44.916417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:44.914200    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7574078-8dcf-4713-9574-c489ef73a943-istio-podinfo\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.015332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.015294    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh56d\" (UniqueName: \"kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-kube-api-access-zh56d\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.015579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.015344    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7574078-8dcf-4713-9574-c489ef73a943-istio-podinfo\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.015579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.015385    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7574078-8dcf-4713-9574-c489ef73a943-istiod-ca-cert\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.015579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.015418    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-istio-token\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.015579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.015489    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-socket\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.015579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.015505    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-envoy\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.015579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.015549    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-data\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.015579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.015577    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-credential-socket\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.016003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.015645    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-certs\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.016281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.016258    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-certs\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.016340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.016312    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-socket\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.016553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.016533    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-data\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.016609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.016582    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-credential-socket\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.017482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.017455    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7574078-8dcf-4713-9574-c489ef73a943-istiod-ca-cert\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.018441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.018415    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-envoy\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.023080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.023057    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7574078-8dcf-4713-9574-c489ef73a943-istio-podinfo\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.031519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.031491    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh56d\" (UniqueName: \"kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-kube-api-access-zh56d\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.032796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.032769    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-istio-token\") pod \"e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") " pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.148418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.147920    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:45.547278 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:31:45.547233    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7574078_8dcf_4713_9574_c489ef73a943.slice/crio-de31f84b03fdd59c740f810a918d9ac5daa4bf3ab26039d623a9f92dbe4f5edc WatchSource:0}: Error finding container de31f84b03fdd59c740f810a918d9ac5daa4bf3ab26039d623a9f92dbe4f5edc: Status 404 returned error can't find the container with id de31f84b03fdd59c740f810a918d9ac5daa4bf3ab26039d623a9f92dbe4f5edc
Oct 30 21:31:45.547507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.547342    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"]
Oct 30 21:31:45.550686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.550654    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:31:45.550780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.550739    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:31:45.550780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.550768    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:31:45.779472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.779436    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" event={"ID":"f7574078-8dcf-4713-9574-c489ef73a943","Type":"ContainerStarted","Data":"0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934"}
Oct 30 21:31:45.779895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.779503    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" event={"ID":"f7574078-8dcf-4713-9574-c489ef73a943","Type":"ContainerStarted","Data":"de31f84b03fdd59c740f810a918d9ac5daa4bf3ab26039d623a9f92dbe4f5edc"}
Oct 30 21:31:45.818803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:45.818751    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" podStartSLOduration=1.818734317 podStartE2EDuration="1.818734317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:45.818638261 +0000 UTC m=+1818.865197407" watchObservedRunningTime="2026-10-30 21:31:45.818734317 +0000 UTC m=+1818.865293462"
Oct 30 21:31:46.108156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.098470    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7663-d7ff7b"]
Oct 30 21:31:46.116189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.111633    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7663-d7ff7b"
Oct 30 21:31:46.116189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.114488    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7663-d7ff7b"]
Oct 30 21:31:46.139414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.117107    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:31:46.139414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.117364    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9lmpp\""
Oct 30 21:31:46.139414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.117541    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:31:46.167783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.154294    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:46.167783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.156328    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 21:31:46.167783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.156409    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 21:31:46.167783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.163140    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcfzw\" (UniqueName: \"kubernetes.io/projected/a1772332-d4d6-420a-9ddd-389de313b881-kube-api-access-xcfzw\") pod \"test-curl-7663-d7ff7b\" (UID: \"a1772332-d4d6-420a-9ddd-389de313b881\") " pod="opendatahub/test-curl-7663-d7ff7b"
Oct 30 21:31:46.265024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.264985    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcfzw\" (UniqueName: \"kubernetes.io/projected/a1772332-d4d6-420a-9ddd-389de313b881-kube-api-access-xcfzw\") pod \"test-curl-7663-d7ff7b\" (UID: \"a1772332-d4d6-420a-9ddd-389de313b881\") " pod="opendatahub/test-curl-7663-d7ff7b"
Oct 30 21:31:46.275475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.275452    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcfzw\" (UniqueName: \"kubernetes.io/projected/a1772332-d4d6-420a-9ddd-389de313b881-kube-api-access-xcfzw\") pod \"test-curl-7663-d7ff7b\" (UID: \"a1772332-d4d6-420a-9ddd-389de313b881\") " pod="opendatahub/test-curl-7663-d7ff7b"
Oct 30 21:31:46.472586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.472556    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7663-d7ff7b"
Oct 30 21:31:46.782017 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:31:46.781979    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1772332_d4d6_420a_9ddd_389de313b881.slice/crio-30fd3aa82d1596d1829408974f9f7cde20cec1e6e358329f9e4fa82943947e72 WatchSource:0}: Error finding container 30fd3aa82d1596d1829408974f9f7cde20cec1e6e358329f9e4fa82943947e72: Status 404 returned error can't find the container with id 30fd3aa82d1596d1829408974f9f7cde20cec1e6e358329f9e4fa82943947e72
Oct 30 21:31:46.782650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:46.782626    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7663-d7ff7b"]
Oct 30 21:31:47.053615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:47.053576    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"]
Oct 30 21:31:47.060759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:47.060737    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-545654d1-qb7jp"]
Oct 30 21:31:47.081599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:47.081565    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96" path="/var/lib/kubelet/pods/fc91c205-89a4-4b6f-8ba4-1edb6d0b7c96/volumes"
Oct 30 21:31:47.154764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:47.154737    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:47.841180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:47.841141    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7663-d7ff7b" event={"ID":"a1772332-d4d6-420a-9ddd-389de313b881","Type":"ContainerStarted","Data":"ecbce9dff498e3b1be8ddfc5d49ff00e272d343ba3da569903fa70ce34b804ea"}
Oct 30 21:31:47.841180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:47.841189    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7663-d7ff7b" event={"ID":"a1772332-d4d6-420a-9ddd-389de313b881","Type":"ContainerStarted","Data":"30fd3aa82d1596d1829408974f9f7cde20cec1e6e358329f9e4fa82943947e72"}
Oct 30 21:31:47.841761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:47.841442    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:47.842558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:47.842538    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:31:47.859997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:47.859955    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7663-d7ff7b" podStartSLOduration=1.859943337 podStartE2EDuration="1.859943337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:47.855197104 +0000 UTC m=+1820.901756465" watchObservedRunningTime="2026-10-30 21:31:47.859943337 +0000 UTC m=+1820.906502482"
Oct 30 21:31:48.920510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:48.920478    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7663-d7ff7b"]
Oct 30 21:31:48.925135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:48.925106    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7663-d7ff7b"]
Oct 30 21:31:49.140473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.140439    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"]
Oct 30 21:31:49.141199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.141186    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a1772332-d4d6-420a-9ddd-389de313b881" containerName="test-curl-7663-d7ff7b"
Oct 30 21:31:49.141250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.141201    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1772332-d4d6-420a-9ddd-389de313b881" containerName="test-curl-7663-d7ff7b"
Oct 30 21:31:49.141445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.141434    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1772332-d4d6-420a-9ddd-389de313b881" containerName="test-curl-7663-d7ff7b"
Oct 30 21:31:49.145453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.145433    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:49.151113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.151087    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:31:49.151263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.151173    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:31:49.154306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.154278    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"]
Oct 30 21:31:49.210756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.210647    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:49.210756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.210715    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:31:49.216318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.216288    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cabdb7f4-49bjt\" (UID: \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:49.216506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.216328    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cabdb7f4-49bjt\" (UID: \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:49.216506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.216492    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w4kc\" (UniqueName: \"kubernetes.io/projected/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-kube-api-access-5w4kc\") pod \"maas-api-revoke-sub-cabdb7f4-49bjt\" (UID: \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:49.318576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.318539    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cabdb7f4-49bjt\" (UID: \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:49.318777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.318588    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cabdb7f4-49bjt\" (UID: \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:49.318883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.318859    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w4kc\" (UniqueName: \"kubernetes.io/projected/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-kube-api-access-5w4kc\") pod \"maas-api-revoke-sub-cabdb7f4-49bjt\" (UID: \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:49.319980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.319336    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cabdb7f4-49bjt\" (UID: \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:49.321450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.321426    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cabdb7f4-49bjt\" (UID: \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:49.332401 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.332372    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w4kc\" (UniqueName: \"kubernetes.io/projected/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-kube-api-access-5w4kc\") pod \"maas-api-revoke-sub-cabdb7f4-49bjt\" (UID: \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:49.465196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.465103    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:49.527370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.527336    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"]
Oct 30 21:31:49.533096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.532669    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.549599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.545543    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-fb2b58-openshift-default-dockercfg-7jgd8\""
Oct 30 21:31:49.582513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.582481    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"]
Oct 30 21:31:49.626985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.626136    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc03a02c-5042-4f3c-be33-4443682c2f6d-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.626985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.626285    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.626985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.626329    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.626985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.626376    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.626985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.626479    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8kl4\" (UniqueName: \"kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-kube-api-access-f8kl4\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.626985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.626552    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.626985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.626645    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.626985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.626870    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.626985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.626991    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.729894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.729180    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.729894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.729225    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc03a02c-5042-4f3c-be33-4443682c2f6d-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.729894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.729339    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.729894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.729382    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.729894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.729481    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.729894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.729558    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8kl4\" (UniqueName: \"kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-kube-api-access-f8kl4\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.729894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.729628    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.729894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.729727    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.729894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.729845    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.729894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.729894    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.730697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.730100    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.730697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.730233    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.730697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.730494    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.731437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.731364    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc03a02c-5042-4f3c-be33-4443682c2f6d-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.732549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.732522    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.736977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.736954    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.751916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.750965    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.751916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.751091    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8kl4\" (UniqueName: \"kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-kube-api-access-f8kl4\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:49.861237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.861197    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7663-d7ff7b" podUID="a1772332-d4d6-420a-9ddd-389de313b881" containerName="test-curl-7663-d7ff7b" containerID="cri-o://ecbce9dff498e3b1be8ddfc5d49ff00e272d343ba3da569903fa70ce34b804ea" gracePeriod=2
Oct 30 21:31:49.883267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.878215    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"]
Oct 30 21:31:49.925418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:49.925081    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:50.261117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.260518    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"
Oct 30 21:31:50.263960 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.262135    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"
Oct 30 21:31:50.611927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.611899    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"]
Oct 30 21:31:50.612748 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:31:50.612725    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc03a02c_5042_4f3c_be33_4443682c2f6d.slice/crio-fd2b6fa9321598741436afb10461eab82e183b7184354685ec2344a409501fc1 WatchSource:0}: Error finding container fd2b6fa9321598741436afb10461eab82e183b7184354685ec2344a409501fc1: Status 404 returned error can't find the container with id fd2b6fa9321598741436afb10461eab82e183b7184354685ec2344a409501fc1
Oct 30 21:31:50.615808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.615771    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:31:50.615900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.615838    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:31:50.615900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.615866    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:31:50.872952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.872865    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt" event={"ID":"c6d7485b-dbf3-4b11-9973-b270bcaceb2e","Type":"ContainerStarted","Data":"a3c63b34f59d3cf3c65e34ad14da3ee2c066f29d6221210035ec5bba49836aa3"}
Oct 30 21:31:50.873224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.873191    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt" event={"ID":"c6d7485b-dbf3-4b11-9973-b270bcaceb2e","Type":"ContainerStarted","Data":"699e0310cf4c441254e93006570a6623464a6f4dba3ed5f5dacb978dde99827c"}
Oct 30 21:31:50.875261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.875230    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" event={"ID":"fc03a02c-5042-4f3c-be33-4443682c2f6d","Type":"ContainerStarted","Data":"a5f85956238ef40fa02cf7d14786d6184a719746e22d129a6dc239b3bed068a4"}
Oct 30 21:31:50.875370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.875283    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" event={"ID":"fc03a02c-5042-4f3c-be33-4443682c2f6d","Type":"ContainerStarted","Data":"fd2b6fa9321598741436afb10461eab82e183b7184354685ec2344a409501fc1"}
Oct 30 21:31:50.913502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.913439    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" podStartSLOduration=1.913419249 podStartE2EDuration="1.913419249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:50.911756388 +0000 UTC m=+1823.958315533" watchObservedRunningTime="2026-10-30 21:31:50.913419249 +0000 UTC m=+1823.959978396"
Oct 30 21:31:50.926856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.926825    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:50.937615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:50.937561    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt" podStartSLOduration=1.937542508 podStartE2EDuration="1.937542508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:50.932846588 +0000 UTC m=+1823.979405741" watchObservedRunningTime="2026-10-30 21:31:50.937542508 +0000 UTC m=+1823.984101671"
Oct 30 21:31:51.909263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:51.909225    4918 generic.go:356] "Generic (PLEG): container finished" podID="c6d7485b-dbf3-4b11-9973-b270bcaceb2e" containerID="a3c63b34f59d3cf3c65e34ad14da3ee2c066f29d6221210035ec5bba49836aa3" exitCode=0
Oct 30 21:31:51.911405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:51.911357    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt" event={"ID":"c6d7485b-dbf3-4b11-9973-b270bcaceb2e","Type":"ContainerDied","Data":"a3c63b34f59d3cf3c65e34ad14da3ee2c066f29d6221210035ec5bba49836aa3"}
Oct 30 21:31:51.954649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:51.954025    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:52.174021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:52.173990    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7663-d7ff7b"
Oct 30 21:31:52.281430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:52.281355    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1772332-d4d6-420a-9ddd-389de313b881-kube-api-access-xcfzw\" (UniqueName: \"kubernetes.io/projected/a1772332-d4d6-420a-9ddd-389de313b881-kube-api-access-xcfzw\") pod \"a1772332-d4d6-420a-9ddd-389de313b881\" (UID: \"a1772332-d4d6-420a-9ddd-389de313b881\") "
Oct 30 21:31:52.283854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:52.283819    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1772332-d4d6-420a-9ddd-389de313b881-kube-api-access-xcfzw" pod "a1772332-d4d6-420a-9ddd-389de313b881" (UID: "a1772332-d4d6-420a-9ddd-389de313b881"). InnerVolumeSpecName "kube-api-access-xcfzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:52.383517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:52.383481    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcfzw\" (UniqueName: \"kubernetes.io/projected/a1772332-d4d6-420a-9ddd-389de313b881-kube-api-access-xcfzw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:52.920783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:52.920748    4918 generic.go:356] "Generic (PLEG): container finished" podID="a1772332-d4d6-420a-9ddd-389de313b881" containerID="ecbce9dff498e3b1be8ddfc5d49ff00e272d343ba3da569903fa70ce34b804ea" exitCode=137
Oct 30 21:31:52.920992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:52.920823    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7663-d7ff7b"
Oct 30 21:31:52.920992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:52.920852    4918 scope.go:122] "RemoveContainer" containerID="ecbce9dff498e3b1be8ddfc5d49ff00e272d343ba3da569903fa70ce34b804ea"
Oct 30 21:31:52.921884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:52.921476    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:52.922645 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:52.922624    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:31:52.954161 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:31:52.954116    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7663-d7ff7b\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="a1772332-d4d6-420a-9ddd-389de313b881" pod="opendatahub/test-curl-7663-d7ff7b"
Oct 30 21:31:52.985968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:52.985934    4918 scope.go:122] "RemoveContainer" containerID="ecbce9dff498e3b1be8ddfc5d49ff00e272d343ba3da569903fa70ce34b804ea"
Oct 30 21:31:52.991804 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:31:52.991764    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbce9dff498e3b1be8ddfc5d49ff00e272d343ba3da569903fa70ce34b804ea\": container with ID starting with ecbce9dff498e3b1be8ddfc5d49ff00e272d343ba3da569903fa70ce34b804ea not found: ID does not exist" containerID="ecbce9dff498e3b1be8ddfc5d49ff00e272d343ba3da569903fa70ce34b804ea"
Oct 30 21:31:52.991983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:52.991824    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbce9dff498e3b1be8ddfc5d49ff00e272d343ba3da569903fa70ce34b804ea"} err="failed to get container status \"ecbce9dff498e3b1be8ddfc5d49ff00e272d343ba3da569903fa70ce34b804ea\": rpc error: code = NotFound desc = could not find container \"ecbce9dff498e3b1be8ddfc5d49ff00e272d343ba3da569903fa70ce34b804ea\": container with ID starting with ecbce9dff498e3b1be8ddfc5d49ff00e272d343ba3da569903fa70ce34b804ea not found: ID does not exist"
Oct 30 21:31:53.225154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.224583    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1772332-d4d6-420a-9ddd-389de313b881" path="/var/lib/kubelet/pods/a1772332-d4d6-420a-9ddd-389de313b881/volumes"
Oct 30 21:31:53.291839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.291816    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:53.404856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.404808    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-kube-api-access-5w4kc\" (UniqueName: \"kubernetes.io/projected/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-kube-api-access-5w4kc\") pod \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\" (UID: \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\") "
Oct 30 21:31:53.405070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.404952    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-openshift-ca\") pod \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\" (UID: \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\") "
Oct 30 21:31:53.405235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.405213    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-xks-ca\") pod \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\" (UID: \"c6d7485b-dbf3-4b11-9973-b270bcaceb2e\") "
Oct 30 21:31:53.405928 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.405368    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-openshift-ca" pod "c6d7485b-dbf3-4b11-9973-b270bcaceb2e" (UID: "c6d7485b-dbf3-4b11-9973-b270bcaceb2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:31:53.406206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.406170    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:53.407607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.407582    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-kube-api-access-5w4kc" pod "c6d7485b-dbf3-4b11-9973-b270bcaceb2e" (UID: "c6d7485b-dbf3-4b11-9973-b270bcaceb2e"). InnerVolumeSpecName "kube-api-access-5w4kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:31:53.408083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.408057    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-xks-ca" pod "c6d7485b-dbf3-4b11-9973-b270bcaceb2e" (UID: "c6d7485b-dbf3-4b11-9973-b270bcaceb2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:31:53.507557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.507489    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:53.507557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.507516    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w4kc\" (UniqueName: \"kubernetes.io/projected/c6d7485b-dbf3-4b11-9973-b270bcaceb2e-kube-api-access-5w4kc\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:31:53.935578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.935502    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"
Oct 30 21:31:53.935578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.935538    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt" event={"ID":"c6d7485b-dbf3-4b11-9973-b270bcaceb2e","Type":"ContainerDied","Data":"699e0310cf4c441254e93006570a6623464a6f4dba3ed5f5dacb978dde99827c"}
Oct 30 21:31:53.935811 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:53.935594    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="699e0310cf4c441254e93006570a6623464a6f4dba3ed5f5dacb978dde99827c"
Oct 30 21:31:54.618933 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.618899    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"]
Oct 30 21:31:54.619850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.619823    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c6d7485b-dbf3-4b11-9973-b270bcaceb2e" containerName="revoke-keys"
Oct 30 21:31:54.619850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.619848    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d7485b-dbf3-4b11-9973-b270bcaceb2e" containerName="revoke-keys"
Oct 30 21:31:54.620135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.620106    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d7485b-dbf3-4b11-9973-b270bcaceb2e" containerName="revoke-keys"
Oct 30 21:31:54.624966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.624946    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"
Oct 30 21:31:54.636759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.636731    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"]
Oct 30 21:31:54.855838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.707547    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-fb2b58\""
Oct 30 21:31:54.855838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.712329    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-fb2b58-dockercfg-xh76c\""
Oct 30 21:31:54.855838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.739416    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9938d41e-a3ab-439c-a2ed-38673adc54c9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6\" (UID: \"9938d41e-a3ab-439c-a2ed-38673adc54c9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"
Oct 30 21:31:54.855838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.739774    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8zq\" (UniqueName: \"kubernetes.io/projected/9938d41e-a3ab-439c-a2ed-38673adc54c9-kube-api-access-pp8zq\") pod \"payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6\" (UID: \"9938d41e-a3ab-439c-a2ed-38673adc54c9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"
Oct 30 21:31:54.882909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.882822    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9938d41e-a3ab-439c-a2ed-38673adc54c9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6\" (UID: \"9938d41e-a3ab-439c-a2ed-38673adc54c9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"
Oct 30 21:31:54.883596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.883568    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8zq\" (UniqueName: \"kubernetes.io/projected/9938d41e-a3ab-439c-a2ed-38673adc54c9-kube-api-access-pp8zq\") pod \"payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6\" (UID: \"9938d41e-a3ab-439c-a2ed-38673adc54c9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"
Oct 30 21:31:54.895129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:54.895099    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9938d41e-a3ab-439c-a2ed-38673adc54c9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6\" (UID: \"9938d41e-a3ab-439c-a2ed-38673adc54c9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"
Oct 30 21:31:55.038808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:55.009574    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8zq\" (UniqueName: \"kubernetes.io/projected/9938d41e-a3ab-439c-a2ed-38673adc54c9-kube-api-access-pp8zq\") pod \"payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6\" (UID: \"9938d41e-a3ab-439c-a2ed-38673adc54c9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"
Oct 30 21:31:55.252702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:55.252631    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"
Oct 30 21:31:55.549191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:55.548865    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"]
Oct 30 21:31:55.549718 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:31:55.549693    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9938d41e_a3ab_439c_a2ed_38673adc54c9.slice/crio-5e78864ef77a7b23b9f28988cf10f875696e2a1e140732d467fda0bb9aab2ed1 WatchSource:0}: Error finding container 5e78864ef77a7b23b9f28988cf10f875696e2a1e140732d467fda0bb9aab2ed1: Status 404 returned error can't find the container with id 5e78864ef77a7b23b9f28988cf10f875696e2a1e140732d467fda0bb9aab2ed1
Oct 30 21:31:56.057668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:56.057633    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6" event={"ID":"9938d41e-a3ab-439c-a2ed-38673adc54c9","Type":"ContainerStarted","Data":"7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1"}
Oct 30 21:31:56.058126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:56.057695    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6" event={"ID":"9938d41e-a3ab-439c-a2ed-38673adc54c9","Type":"ContainerStarted","Data":"5e78864ef77a7b23b9f28988cf10f875696e2a1e140732d467fda0bb9aab2ed1"}
Oct 30 21:31:56.058126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:56.057723    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"
Oct 30 21:31:56.073934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:56.073895    4918 generic.go:356] "Generic (PLEG): container finished" podID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerID="3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03" exitCode=0
Oct 30 21:31:56.074133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:56.073938    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" event={"ID":"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3","Type":"ContainerDied","Data":"3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03"}
Oct 30 21:31:56.074497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:56.074470    4918 scope.go:122] "RemoveContainer" containerID="3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03"
Oct 30 21:31:56.076311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:56.076281    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:31:56.076440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:56.076347    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:31:56.076440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:56.076382    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:31:56.198695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:56.186570    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6" podStartSLOduration=2.1865486179999998 podStartE2EDuration="2.186548618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:56.106053879 +0000 UTC m=+1829.152613044" watchObservedRunningTime="2026-10-30 21:31:56.186548618 +0000 UTC m=+1829.233107764"
Oct 30 21:31:57.025131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:57.025099    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"]
Oct 30 21:31:57.029843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:57.029820    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df686646-7klp5"]
Oct 30 21:31:57.081431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:57.081401    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea44253-7753-4a15-aae6-70d5d85da18b" path="/var/lib/kubelet/pods/cea44253-7753-4a15-aae6-70d5d85da18b/volumes"
Oct 30 21:31:57.086417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:57.086374    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" event={"ID":"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3","Type":"ContainerStarted","Data":"98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc"}
Oct 30 21:31:57.818152 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:57.818120    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:31:57.819953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:57.819924    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 21:31:57.820103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:57.819981    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 21:31:58.818734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:58.818704    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 21:31:58.819151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:58.818754    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 21:31:59.241613 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:31:59.241541    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:31:59.285327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.285297    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:31:59.416987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.416954    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"]
Oct 30 21:31:59.424855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.424827    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:31:59.430356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.430329    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"]
Oct 30 21:31:59.430841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.430823    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:31:59.431595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.431576    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:31:59.431716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.431577    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:31:59.432671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.432654    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:31:59.511489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.511346    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmxs6\" (UniqueName: \"kubernetes.io/projected/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-kube-api-access-wmxs6\") pod \"maas-api-revoke-sub-5af51fca-4nxnz\" (UID: \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:31:59.511692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.511657    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5af51fca-4nxnz\" (UID: \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:31:59.511932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.511913    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5af51fca-4nxnz\" (UID: \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:31:59.613684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.613651    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5af51fca-4nxnz\" (UID: \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:31:59.613879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.613775    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmxs6\" (UniqueName: \"kubernetes.io/projected/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-kube-api-access-wmxs6\") pod \"maas-api-revoke-sub-5af51fca-4nxnz\" (UID: \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:31:59.613967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.613892    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5af51fca-4nxnz\" (UID: \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:31:59.614712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.614644    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5af51fca-4nxnz\" (UID: \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:31:59.616365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.616344    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5af51fca-4nxnz\" (UID: \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:31:59.623711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.623685    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmxs6\" (UniqueName: \"kubernetes.io/projected/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-kube-api-access-wmxs6\") pod \"maas-api-revoke-sub-5af51fca-4nxnz\" (UID: \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:31:59.747528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.747495    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:31:59.818810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.818778    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 21:31:59.819327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:31:59.818822    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 21:32:00.257940 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:32:00.257912    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7bb69c_cafa_4838_b17d_39aec6e27a0e.slice/crio-6fee011886b17798d1a4e4f049e85dbf15048aa7ca42b9a0ebfc5fb604c838ea WatchSource:0}: Error finding container 6fee011886b17798d1a4e4f049e85dbf15048aa7ca42b9a0ebfc5fb604c838ea: Status 404 returned error can't find the container with id 6fee011886b17798d1a4e4f049e85dbf15048aa7ca42b9a0ebfc5fb604c838ea
Oct 30 21:32:00.258278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:00.258257    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"]
Oct 30 21:32:00.818504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:00.818475    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 21:32:00.818626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:00.818534    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 21:32:01.232189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:01.232155    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz" event={"ID":"3a7bb69c-cafa-4838-b17d-39aec6e27a0e","Type":"ContainerStarted","Data":"dd8a1e3a7506d20bce298c844be5acaa3c3ca5754475bd65a8879959e77ecad6"}
Oct 30 21:32:01.232634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:01.232214    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz" event={"ID":"3a7bb69c-cafa-4838-b17d-39aec6e27a0e","Type":"ContainerStarted","Data":"6fee011886b17798d1a4e4f049e85dbf15048aa7ca42b9a0ebfc5fb604c838ea"}
Oct 30 21:32:01.256735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:01.256685    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz" podStartSLOduration=2.2566682399999998 podStartE2EDuration="2.25666824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:01.254462271 +0000 UTC m=+1834.301021428" watchObservedRunningTime="2026-10-30 21:32:01.25666824 +0000 UTC m=+1834.303227390"
Oct 30 21:32:01.762740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:01.762650    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"]
Oct 30 21:32:01.762924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:01.762852    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" podUID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" containerName="istio-proxy" containerID="cri-o://5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411" gracePeriod=30
Oct 30 21:32:01.828182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:01.828149    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 21:32:01.828510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:01.828479    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 21:32:02.243169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:02.243137    4918 generic.go:356] "Generic (PLEG): container finished" podID="3a7bb69c-cafa-4838-b17d-39aec6e27a0e" containerID="dd8a1e3a7506d20bce298c844be5acaa3c3ca5754475bd65a8879959e77ecad6" exitCode=0
Oct 30 21:32:02.243697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:02.243232    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz" event={"ID":"3a7bb69c-cafa-4838-b17d-39aec6e27a0e","Type":"ContainerDied","Data":"dd8a1e3a7506d20bce298c844be5acaa3c3ca5754475bd65a8879959e77ecad6"}
Oct 30 21:32:02.818980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:02.818937    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Oct 30 21:32:02.819206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:02.818992    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Oct 30 21:32:03.431340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:03.431316    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:32:03.461851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:03.461826    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-kube-api-access-wmxs6\" (UniqueName: \"kubernetes.io/projected/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-kube-api-access-wmxs6\") pod \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\" (UID: \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\") "
Oct 30 21:32:03.462010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:03.461985    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-xks-ca\") pod \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\" (UID: \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\") "
Oct 30 21:32:03.462109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:03.462098    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-openshift-ca\") pod \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\" (UID: \"3a7bb69c-cafa-4838-b17d-39aec6e27a0e\") "
Oct 30 21:32:03.462623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:03.462533    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-openshift-ca" pod "3a7bb69c-cafa-4838-b17d-39aec6e27a0e" (UID: "3a7bb69c-cafa-4838-b17d-39aec6e27a0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:03.462623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:03.462643    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:03.464244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:03.464212    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-xks-ca" pod "3a7bb69c-cafa-4838-b17d-39aec6e27a0e" (UID: "3a7bb69c-cafa-4838-b17d-39aec6e27a0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:03.464346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:03.464302    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-kube-api-access-wmxs6" pod "3a7bb69c-cafa-4838-b17d-39aec6e27a0e" (UID: "3a7bb69c-cafa-4838-b17d-39aec6e27a0e"). InnerVolumeSpecName "kube-api-access-wmxs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:03.564373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:03.564300    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:03.564373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:03.564327    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmxs6\" (UniqueName: \"kubernetes.io/projected/3a7bb69c-cafa-4838-b17d-39aec6e27a0e-kube-api-access-wmxs6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:03.824050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:03.823971    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:32:03.825212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:03.825172    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:32:04.306621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:04.306579    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz" event={"ID":"3a7bb69c-cafa-4838-b17d-39aec6e27a0e","Type":"ContainerDied","Data":"6fee011886b17798d1a4e4f049e85dbf15048aa7ca42b9a0ebfc5fb604c838ea"}
Oct 30 21:32:04.307066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:04.306997    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fee011886b17798d1a4e4f049e85dbf15048aa7ca42b9a0ebfc5fb604c838ea"
Oct 30 21:32:04.307251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:04.306924    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"
Oct 30 21:32:05.661537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:05.661491    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:05.662838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:05.662351    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:05.664080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:05.662403    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:32:05.664892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:05.662438    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:05.725518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:05.725473    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:05.823517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:05.823473    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:05.886549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:05.886498    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:05.923626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:05.923501    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:05.957546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:05.957497    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:05.966523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:05.966479    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:05.993533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:05.993482    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:06.049294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:06.049254    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:06.077760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:06.077716    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:32:07.124112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.124088    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"
Oct 30 21:32:07.154169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.154147    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:32:07.357418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.332033    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-envoy\") pod \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") "
Oct 30 21:32:07.358499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.358468    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-data\") pod \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") "
Oct 30 21:32:07.364976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.360694    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-data" pod "fa7e7554-6fef-4b37-ac7d-1fce50662a0e" (UID: "fa7e7554-6fef-4b37-ac7d-1fce50662a0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:07.364976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.364534    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-token\" (UniqueName: \"kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-token\") pod \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") "
Oct 30 21:32:07.364976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.364619    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-socket\") pod \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") "
Oct 30 21:32:07.364976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.364652    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istiod-ca-cert\") pod \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") "
Oct 30 21:32:07.364976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.364809    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-kube-api-access-bm2x4\" (UniqueName: \"kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-kube-api-access-bm2x4\") pod \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") "
Oct 30 21:32:07.364976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.364881    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-certs\") pod \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") "
Oct 30 21:32:07.365417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.365067    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-credential-socket\") pod \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") "
Oct 30 21:32:07.365417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.365101    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-podinfo\") pod \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\" (UID: \"fa7e7554-6fef-4b37-ac7d-1fce50662a0e\") "
Oct 30 21:32:07.366381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.366286    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-certs" pod "fa7e7554-6fef-4b37-ac7d-1fce50662a0e" (UID: "fa7e7554-6fef-4b37-ac7d-1fce50662a0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:07.372544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.372514    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-credential-socket" pod "fa7e7554-6fef-4b37-ac7d-1fce50662a0e" (UID: "fa7e7554-6fef-4b37-ac7d-1fce50662a0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:07.372892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.372875    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-socket" pod "fa7e7554-6fef-4b37-ac7d-1fce50662a0e" (UID: "fa7e7554-6fef-4b37-ac7d-1fce50662a0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:07.373262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.373246    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istiod-ca-cert" pod "fa7e7554-6fef-4b37-ac7d-1fce50662a0e" (UID: "fa7e7554-6fef-4b37-ac7d-1fce50662a0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:07.380586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.380545    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-envoy" pod "fa7e7554-6fef-4b37-ac7d-1fce50662a0e" (UID: "fa7e7554-6fef-4b37-ac7d-1fce50662a0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:07.383456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.383373    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-kube-api-access-bm2x4" pod "fa7e7554-6fef-4b37-ac7d-1fce50662a0e" (UID: "fa7e7554-6fef-4b37-ac7d-1fce50662a0e"). InnerVolumeSpecName "kube-api-access-bm2x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:07.383950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.383921    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-token" pod "fa7e7554-6fef-4b37-ac7d-1fce50662a0e" (UID: "fa7e7554-6fef-4b37-ac7d-1fce50662a0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:07.384521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.384489    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-podinfo" pod "fa7e7554-6fef-4b37-ac7d-1fce50662a0e" (UID: "fa7e7554-6fef-4b37-ac7d-1fce50662a0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:07.420481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.418244    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:07.420481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.418469    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:07.420481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.418502    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:07.420481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.418526    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:07.420481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.418563    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:07.420481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.418579    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm2x4\" (UniqueName: \"kubernetes.io/projected/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-kube-api-access-bm2x4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:07.420481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.418597    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:07.420481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.418630    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:07.420481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.418646    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7e7554-6fef-4b37-ac7d-1fce50662a0e-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:07.434448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.426562    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"]
Oct 30 21:32:07.446240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.446204    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" containerName="istio-proxy"
Oct 30 21:32:07.446528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.446509    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" containerName="istio-proxy"
Oct 30 21:32:07.446678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.446665    4918 cpu_manager.go:414] "containerMap: removing container" podUID="3a7bb69c-cafa-4838-b17d-39aec6e27a0e" containerName="revoke-keys"
Oct 30 21:32:07.446778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.446765    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7bb69c-cafa-4838-b17d-39aec6e27a0e" containerName="revoke-keys"
Oct 30 21:32:07.447098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.447082    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" containerName="istio-proxy"
Oct 30 21:32:07.447214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.447202    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7bb69c-cafa-4838-b17d-39aec6e27a0e" containerName="revoke-keys"
Oct 30 21:32:07.452529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.452505    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"]
Oct 30 21:32:07.454582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.454559    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:07.460658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.460636    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:32:07.460811 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.460685    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:32:07.460903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.460761    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:32:07.460992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.460823    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:32:07.511790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.511752    4918 generic.go:356] "Generic (PLEG): container finished" podID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" containerID="5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411" exitCode=0
Oct 30 21:32:07.511947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.511883    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" event={"ID":"fa7e7554-6fef-4b37-ac7d-1fce50662a0e","Type":"ContainerDied","Data":"5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411"}
Oct 30 21:32:07.511947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.511938    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk" event={"ID":"fa7e7554-6fef-4b37-ac7d-1fce50662a0e","Type":"ContainerDied","Data":"b44475e44c56a9d0d8c3452ac435182c8e376a02a411997ce10ae43f068da815"}
Oct 30 21:32:07.512058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.511969    4918 scope.go:122] "RemoveContainer" containerID="5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411"
Oct 30 21:32:07.512340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.512144    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"
Oct 30 21:32:07.519840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.519813    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6215f003-lgbzh\" (UID: \"b1d645ed-64b5-4c16-9338-b2eacc76be08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:07.520168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.520149    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6215f003-lgbzh\" (UID: \"b1d645ed-64b5-4c16-9338-b2eacc76be08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:07.520358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.520340    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgmsg\" (UniqueName: \"kubernetes.io/projected/b1d645ed-64b5-4c16-9338-b2eacc76be08-kube-api-access-hgmsg\") pod \"maas-api-revoke-sub-6215f003-lgbzh\" (UID: \"b1d645ed-64b5-4c16-9338-b2eacc76be08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:07.526496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.526479    4918 scope.go:122] "RemoveContainer" containerID="5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411"
Oct 30 21:32:07.526802 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:32:07.526715    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411\": container with ID starting with 5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411 not found: ID does not exist" containerID="5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411"
Oct 30 21:32:07.526802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.526739    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411"} err="failed to get container status \"5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411\": rpc error: code = NotFound desc = could not find container \"5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411\": container with ID starting with 5c79af1cb6129e23df3647d6dca7eba1a1c139fd55c1fbfc6d25f5f9e4124411 not found: ID does not exist"
Oct 30 21:32:07.543675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.543655    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"]
Oct 30 21:32:07.550485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.550465    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-grpnk"]
Oct 30 21:32:07.622278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.622190    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6215f003-lgbzh\" (UID: \"b1d645ed-64b5-4c16-9338-b2eacc76be08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:07.622278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.622253    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgmsg\" (UniqueName: \"kubernetes.io/projected/b1d645ed-64b5-4c16-9338-b2eacc76be08-kube-api-access-hgmsg\") pod \"maas-api-revoke-sub-6215f003-lgbzh\" (UID: \"b1d645ed-64b5-4c16-9338-b2eacc76be08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:07.622551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.622406    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6215f003-lgbzh\" (UID: \"b1d645ed-64b5-4c16-9338-b2eacc76be08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:07.622931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.622907    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6215f003-lgbzh\" (UID: \"b1d645ed-64b5-4c16-9338-b2eacc76be08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:07.624952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.624933    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6215f003-lgbzh\" (UID: \"b1d645ed-64b5-4c16-9338-b2eacc76be08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:07.630836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.630818    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgmsg\" (UniqueName: \"kubernetes.io/projected/b1d645ed-64b5-4c16-9338-b2eacc76be08-kube-api-access-hgmsg\") pod \"maas-api-revoke-sub-6215f003-lgbzh\" (UID: \"b1d645ed-64b5-4c16-9338-b2eacc76be08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:07.778540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:07.778509    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:08.354475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:08.352260    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"]
Oct 30 21:32:08.355383 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:32:08.355359    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d645ed_64b5_4c16_9338_b2eacc76be08.slice/crio-8383c1bca3c5dbd13f842a6d68a4cc4ff27b863a4d47623005087c0af2ad9581 WatchSource:0}: Error finding container 8383c1bca3c5dbd13f842a6d68a4cc4ff27b863a4d47623005087c0af2ad9581: Status 404 returned error can't find the container with id 8383c1bca3c5dbd13f842a6d68a4cc4ff27b863a4d47623005087c0af2ad9581
Oct 30 21:32:08.522202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:08.522165    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh" event={"ID":"b1d645ed-64b5-4c16-9338-b2eacc76be08","Type":"ContainerStarted","Data":"8383c1bca3c5dbd13f842a6d68a4cc4ff27b863a4d47623005087c0af2ad9581"}
Oct 30 21:32:09.080295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:09.080262    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa7e7554-6fef-4b37-ac7d-1fce50662a0e" path="/var/lib/kubelet/pods/fa7e7554-6fef-4b37-ac7d-1fce50662a0e/volumes"
Oct 30 21:32:09.877743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:09.877639    4918 generic.go:356] "Generic (PLEG): container finished" podID="b1d645ed-64b5-4c16-9338-b2eacc76be08" containerID="d0a0bbdbca96b709ed1efaa0193522f16709aee2f795580bfe252636edd12194" exitCode=0
Oct 30 21:32:09.877743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:09.877717    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh" event={"ID":"b1d645ed-64b5-4c16-9338-b2eacc76be08","Type":"ContainerDied","Data":"d0a0bbdbca96b709ed1efaa0193522f16709aee2f795580bfe252636edd12194"}
Oct 30 21:32:11.136804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.136779    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:11.276210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.276171    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-xks-ca\") pod \"b1d645ed-64b5-4c16-9338-b2eacc76be08\" (UID: \"b1d645ed-64b5-4c16-9338-b2eacc76be08\") "
Oct 30 21:32:11.276629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.276602    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1d645ed-64b5-4c16-9338-b2eacc76be08-kube-api-access-hgmsg\" (UniqueName: \"kubernetes.io/projected/b1d645ed-64b5-4c16-9338-b2eacc76be08-kube-api-access-hgmsg\") pod \"b1d645ed-64b5-4c16-9338-b2eacc76be08\" (UID: \"b1d645ed-64b5-4c16-9338-b2eacc76be08\") "
Oct 30 21:32:11.276892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.276668    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-openshift-ca\") pod \"b1d645ed-64b5-4c16-9338-b2eacc76be08\" (UID: \"b1d645ed-64b5-4c16-9338-b2eacc76be08\") "
Oct 30 21:32:11.283585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.277674    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-openshift-ca" pod "b1d645ed-64b5-4c16-9338-b2eacc76be08" (UID: "b1d645ed-64b5-4c16-9338-b2eacc76be08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:11.283585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.277849    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:11.283585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.279955    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-xks-ca" pod "b1d645ed-64b5-4c16-9338-b2eacc76be08" (UID: "b1d645ed-64b5-4c16-9338-b2eacc76be08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:11.320704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.311599    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d645ed-64b5-4c16-9338-b2eacc76be08-kube-api-access-hgmsg" pod "b1d645ed-64b5-4c16-9338-b2eacc76be08" (UID: "b1d645ed-64b5-4c16-9338-b2eacc76be08"). InnerVolumeSpecName "kube-api-access-hgmsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:11.379268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.379239    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1d645ed-64b5-4c16-9338-b2eacc76be08-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:11.379268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.379269    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgmsg\" (UniqueName: \"kubernetes.io/projected/b1d645ed-64b5-4c16-9338-b2eacc76be08-kube-api-access-hgmsg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:11.978936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.978892    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh" event={"ID":"b1d645ed-64b5-4c16-9338-b2eacc76be08","Type":"ContainerDied","Data":"8383c1bca3c5dbd13f842a6d68a4cc4ff27b863a4d47623005087c0af2ad9581"}
Oct 30 21:32:11.979483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.979462    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8383c1bca3c5dbd13f842a6d68a4cc4ff27b863a4d47623005087c0af2ad9581"
Oct 30 21:32:11.979664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:11.979404    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"
Oct 30 21:32:14.142679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.142639    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"]
Oct 30 21:32:14.144542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.144513    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b1d645ed-64b5-4c16-9338-b2eacc76be08" containerName="revoke-keys"
Oct 30 21:32:14.144721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.144707    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1d645ed-64b5-4c16-9338-b2eacc76be08" containerName="revoke-keys"
Oct 30 21:32:14.145073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.145055    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1d645ed-64b5-4c16-9338-b2eacc76be08" containerName="revoke-keys"
Oct 30 21:32:14.199323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.198749    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.307854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.307823    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-06bb0f-openshift-default-dockercfg-r6whw\""
Oct 30 21:32:14.327547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.327513    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"]
Oct 30 21:32:14.528068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.528006    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-certs\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.528505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.528478    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-envoy\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.528812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.528776    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtl75\" (UniqueName: \"kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-kube-api-access-dtl75\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.529015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.528996    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f22573-6b3d-467b-978c-d37e79634546-istio-podinfo\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.529218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.529180    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-credential-socket\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.529427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.529379    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f22573-6b3d-467b-978c-d37e79634546-istiod-ca-cert\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.529585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.529561    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-istio-token\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.529708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.529649    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-socket\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.529775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.529720    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-data\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.631588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.631541    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-socket\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.631751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.631646    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-data\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.631841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.631823    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-certs\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.631943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.631919    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-envoy\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.632096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.632002    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtl75\" (UniqueName: \"kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-kube-api-access-dtl75\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.632096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.632006    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-socket\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.632096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.632020    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-data\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.632096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.632044    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f22573-6b3d-467b-978c-d37e79634546-istio-podinfo\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.632096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.632088    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-credential-socket\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.632377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.632165    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f22573-6b3d-467b-978c-d37e79634546-istiod-ca-cert\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.632377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.632198    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-certs\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.632377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.632212    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-istio-token\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.632654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.632629    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-credential-socket\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.632940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.632885    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f22573-6b3d-467b-978c-d37e79634546-istiod-ca-cert\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.634769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.634736    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-envoy\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.634895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.634874    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f22573-6b3d-467b-978c-d37e79634546-istio-podinfo\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.657243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.657210    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtl75\" (UniqueName: \"kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-kube-api-access-dtl75\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.661825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.661792    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-istio-token\") pod \"e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") " pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:14.974466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:14.963759    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:15.635894 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:32:15.635855    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f22573_6b3d_467b_978c_d37e79634546.slice/crio-e02c5e5e72a706054b7d690a290b172dcec9df4c4074d48347873ce82cdaa941 WatchSource:0}: Error finding container e02c5e5e72a706054b7d690a290b172dcec9df4c4074d48347873ce82cdaa941: Status 404 returned error can't find the container with id e02c5e5e72a706054b7d690a290b172dcec9df4c4074d48347873ce82cdaa941
Oct 30 21:32:15.636347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:15.636160    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"]
Oct 30 21:32:15.639595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:15.639558    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:32:15.639685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:15.639653    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:32:15.639727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:15.639701    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:32:16.196322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:16.196278    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" event={"ID":"f0f22573-6b3d-467b-978c-d37e79634546","Type":"ContainerStarted","Data":"e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557"}
Oct 30 21:32:16.196564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:16.196343    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" event={"ID":"f0f22573-6b3d-467b-978c-d37e79634546","Type":"ContainerStarted","Data":"e02c5e5e72a706054b7d690a290b172dcec9df4c4074d48347873ce82cdaa941"}
Oct 30 21:32:16.220237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:16.220169    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" podStartSLOduration=2.220149578 podStartE2EDuration="2.220149578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:16.217949062 +0000 UTC m=+1849.264508210" watchObservedRunningTime="2026-10-30 21:32:16.220149578 +0000 UTC m=+1849.266708724"
Oct 30 21:32:16.964193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:16.964160    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:16.966183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:16.966158    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 21:32:16.966319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:16.966211    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" podUID="f0f22573-6b3d-467b-978c-d37e79634546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 21:32:17.572249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.572198    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"]
Oct 30 21:32:17.572453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.572426    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" podUID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" containerName="istio-proxy" containerID="cri-o://a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2" gracePeriod=30
Oct 30 21:32:17.671628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.671597    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"]
Oct 30 21:32:17.677432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.677406    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:17.733417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.730762    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:32:17.734450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.734419    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"]
Oct 30 21:32:17.734746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.734721    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:32:17.734897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.734886    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:32:17.735047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.735037    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:32:17.739848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.739818    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e4060f4-hd44l\" (UID: \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:17.739959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.739879    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnx88\" (UniqueName: \"kubernetes.io/projected/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-kube-api-access-dnx88\") pod \"maas-api-revoke-sub-9e4060f4-hd44l\" (UID: \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:17.740082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.740065    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e4060f4-hd44l\" (UID: \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:17.841544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.841429    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e4060f4-hd44l\" (UID: \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:17.841544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.841507    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnx88\" (UniqueName: \"kubernetes.io/projected/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-kube-api-access-dnx88\") pod \"maas-api-revoke-sub-9e4060f4-hd44l\" (UID: \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:17.841788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.841728    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e4060f4-hd44l\" (UID: \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:17.842354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.842334    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e4060f4-hd44l\" (UID: \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:17.844249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.844222    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e4060f4-hd44l\" (UID: \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:17.857802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.857776    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnx88\" (UniqueName: \"kubernetes.io/projected/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-kube-api-access-dnx88\") pod \"maas-api-revoke-sub-9e4060f4-hd44l\" (UID: \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:17.964876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.964847    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 21:32:17.985277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:17.964927    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" podUID="f0f22573-6b3d-467b-978c-d37e79634546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 21:32:18.053321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:18.053286    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:18.371856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:18.371757    4918 generic.go:356] "Generic (PLEG): container finished" podID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerID="fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317" exitCode=0
Oct 30 21:32:18.373165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:18.373127    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" event={"ID":"28d7835c-782d-400b-8c1e-3a4654fd8a12","Type":"ContainerDied","Data":"fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317"}
Oct 30 21:32:18.373596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:18.373577    4918 scope.go:122] "RemoveContainer" containerID="fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317"
Oct 30 21:32:18.375567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:18.375526    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:32:18.375684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:18.375620    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:32:18.375684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:18.375668    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:32:18.583156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:18.583125    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"]
Oct 30 21:32:18.583449 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:32:18.583415    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebbec47_f69f_4f8a_997f_4b1be1120e4c.slice/crio-9b21e212f9db8b38e83cd80cc5d59cf09172f7255b181c387d08b150e01e7392 WatchSource:0}: Error finding container 9b21e212f9db8b38e83cd80cc5d59cf09172f7255b181c387d08b150e01e7392: Status 404 returned error can't find the container with id 9b21e212f9db8b38e83cd80cc5d59cf09172f7255b181c387d08b150e01e7392
Oct 30 21:32:18.980469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:18.966324    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 21:32:18.980469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:18.966431    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" podUID="f0f22573-6b3d-467b-978c-d37e79634546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 21:32:19.384132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:19.384099    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" event={"ID":"28d7835c-782d-400b-8c1e-3a4654fd8a12","Type":"ContainerStarted","Data":"9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3"}
Oct 30 21:32:19.411422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:19.405210    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l" event={"ID":"0ebbec47-f69f-4f8a-997f-4b1be1120e4c","Type":"ContainerStarted","Data":"36af050bba971af0c5bf48eb78dff42ded839ab2a2007b58ebf5b00203af28af"}
Oct 30 21:32:19.411422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:19.405276    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l" event={"ID":"0ebbec47-f69f-4f8a-997f-4b1be1120e4c","Type":"ContainerStarted","Data":"9b21e212f9db8b38e83cd80cc5d59cf09172f7255b181c387d08b150e01e7392"}
Oct 30 21:32:19.460996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:19.460934    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l" podStartSLOduration=2.460908492 podStartE2EDuration="2.460908492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:19.45734706 +0000 UTC m=+1852.503906212" watchObservedRunningTime="2026-10-30 21:32:19.460908492 +0000 UTC m=+1852.507467628"
Oct 30 21:32:19.494603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:19.494566    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:32:19.517536 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:19.517491    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 21:32:19.517848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:19.517815    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 21:32:19.964721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:19.964686    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 21:32:19.965001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:19.964969    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" podUID="f0f22573-6b3d-467b-978c-d37e79634546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 21:32:20.083441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:20.070804    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:32:20.477160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:20.477119    4918 generic.go:356] "Generic (PLEG): container finished" podID="0ebbec47-f69f-4f8a-997f-4b1be1120e4c" containerID="36af050bba971af0c5bf48eb78dff42ded839ab2a2007b58ebf5b00203af28af" exitCode=0
Oct 30 21:32:20.478558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:20.478526    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l" event={"ID":"0ebbec47-f69f-4f8a-997f-4b1be1120e4c","Type":"ContainerDied","Data":"36af050bba971af0c5bf48eb78dff42ded839ab2a2007b58ebf5b00203af28af"}
Oct 30 21:32:20.495853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:20.495817    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 21:32:20.641538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:20.641489    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 21:32:20.964969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:20.964877    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 21:32:20.964969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:20.964937    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" podUID="f0f22573-6b3d-467b-978c-d37e79634546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 21:32:21.508715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:21.508253    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 21:32:21.508715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:21.508319    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 21:32:21.857638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:21.857613    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:21.949417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:21.934922    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-openshift-ca\") pod \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\" (UID: \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\") "
Oct 30 21:32:21.949417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:21.935204    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-kube-api-access-dnx88\" (UniqueName: \"kubernetes.io/projected/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-kube-api-access-dnx88\") pod \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\" (UID: \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\") "
Oct 30 21:32:21.949417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:21.935252    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-xks-ca\") pod \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\" (UID: \"0ebbec47-f69f-4f8a-997f-4b1be1120e4c\") "
Oct 30 21:32:21.969878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:21.962098    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-openshift-ca" pod "0ebbec47-f69f-4f8a-997f-4b1be1120e4c" (UID: "0ebbec47-f69f-4f8a-997f-4b1be1120e4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:21.969878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:21.962670    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-xks-ca" pod "0ebbec47-f69f-4f8a-997f-4b1be1120e4c" (UID: "0ebbec47-f69f-4f8a-997f-4b1be1120e4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:32:21.974429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:21.973029    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 21:32:21.974429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:21.973085    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" podUID="f0f22573-6b3d-467b-978c-d37e79634546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 21:32:21.992902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:21.992867    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-kube-api-access-dnx88" pod "0ebbec47-f69f-4f8a-997f-4b1be1120e4c" (UID: "0ebbec47-f69f-4f8a-997f-4b1be1120e4c"). InnerVolumeSpecName "kube-api-access-dnx88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:22.041448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:22.039101    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:22.041448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:22.039140    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnx88\" (UniqueName: \"kubernetes.io/projected/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-kube-api-access-dnx88\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:22.041448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:22.039158    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebbec47-f69f-4f8a-997f-4b1be1120e4c-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:22.545419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:22.544938    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 21:32:22.545419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:22.545025    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 21:32:22.546686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:22.546652    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l" event={"ID":"0ebbec47-f69f-4f8a-997f-4b1be1120e4c","Type":"ContainerDied","Data":"9b21e212f9db8b38e83cd80cc5d59cf09172f7255b181c387d08b150e01e7392"}
Oct 30 21:32:22.546833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:22.546704    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b21e212f9db8b38e83cd80cc5d59cf09172f7255b181c387d08b150e01e7392"
Oct 30 21:32:22.546833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:22.546757    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"
Oct 30 21:32:22.971696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:22.964826    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Oct 30 21:32:22.971696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:22.964879    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" podUID="f0f22573-6b3d-467b-978c-d37e79634546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Oct 30 21:32:23.248287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.247996    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:32:23.348241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.348204    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-envoy\") pod \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") "
Oct 30 21:32:23.348462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.348446    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istiod-ca-cert\") pod \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") "
Oct 30 21:32:23.348616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.348595    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-certs\") pod \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") "
Oct 30 21:32:23.348720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.348703    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-credential-socket\") pod \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") "
Oct 30 21:32:23.348825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.348795    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-token\" (UniqueName: \"kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-token\") pod \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") "
Oct 30 21:32:23.348946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.348930    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-data\") pod \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") "
Oct 30 21:32:23.349013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.348984    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-socket\") pod \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") "
Oct 30 21:32:23.349013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.348989    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istiod-ca-cert" pod "e957b4da-6819-42ee-9afa-e89cef9c4bfd" (UID: "e957b4da-6819-42ee-9afa-e89cef9c4bfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:23.349158 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.349135    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-kube-api-access-ngtqn\" (UniqueName: \"kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-kube-api-access-ngtqn\") pod \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") "
Oct 30 21:32:23.349296 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.349274    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-socket" pod "e957b4da-6819-42ee-9afa-e89cef9c4bfd" (UID: "e957b4da-6819-42ee-9afa-e89cef9c4bfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:23.349383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.349289    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-podinfo\") pod \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\" (UID: \"e957b4da-6819-42ee-9afa-e89cef9c4bfd\") "
Oct 30 21:32:23.349560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.349522    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-certs" pod "e957b4da-6819-42ee-9afa-e89cef9c4bfd" (UID: "e957b4da-6819-42ee-9afa-e89cef9c4bfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:23.349757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.349734    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-credential-socket" pod "e957b4da-6819-42ee-9afa-e89cef9c4bfd" (UID: "e957b4da-6819-42ee-9afa-e89cef9c4bfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:23.349924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.349899    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-data" pod "e957b4da-6819-42ee-9afa-e89cef9c4bfd" (UID: "e957b4da-6819-42ee-9afa-e89cef9c4bfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:23.350208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.350186    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:23.350276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.350214    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:23.350276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.350230    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:23.350276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.350244    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:23.350276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.350257    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:23.354317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.354291    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-token" pod "e957b4da-6819-42ee-9afa-e89cef9c4bfd" (UID: "e957b4da-6819-42ee-9afa-e89cef9c4bfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:23.354421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.354338    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-kube-api-access-ngtqn" pod "e957b4da-6819-42ee-9afa-e89cef9c4bfd" (UID: "e957b4da-6819-42ee-9afa-e89cef9c4bfd"). InnerVolumeSpecName "kube-api-access-ngtqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:23.354504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.354483    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-envoy" pod "e957b4da-6819-42ee-9afa-e89cef9c4bfd" (UID: "e957b4da-6819-42ee-9afa-e89cef9c4bfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:23.354602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.354581    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-podinfo" pod "e957b4da-6819-42ee-9afa-e89cef9c4bfd" (UID: "e957b4da-6819-42ee-9afa-e89cef9c4bfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:23.452010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.451972    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngtqn\" (UniqueName: \"kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-kube-api-access-ngtqn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:23.452010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.452002    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:23.452010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.452016    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:23.452370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.452025    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e957b4da-6819-42ee-9afa-e89cef9c4bfd-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:23.500473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.492796    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 21:32:23.500473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.492874    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 21:32:23.612418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.610146    4918 generic.go:356] "Generic (PLEG): container finished" podID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" containerID="a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2" exitCode=0
Oct 30 21:32:23.612418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.610438    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" event={"ID":"e957b4da-6819-42ee-9afa-e89cef9c4bfd","Type":"ContainerDied","Data":"a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2"}
Oct 30 21:32:23.612418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.610491    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn" event={"ID":"e957b4da-6819-42ee-9afa-e89cef9c4bfd","Type":"ContainerDied","Data":"d4b791873b98a6bfea8a38c77dd438dc1b492264072b7dcc0f37b0a22be96673"}
Oct 30 21:32:23.612418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.610521    4918 scope.go:122] "RemoveContainer" containerID="a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2"
Oct 30 21:32:23.612418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.610700    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"
Oct 30 21:32:23.641881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.641649    4918 scope.go:122] "RemoveContainer" containerID="a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2"
Oct 30 21:32:23.642082 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:32:23.642049    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2\": container with ID starting with a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2 not found: ID does not exist" containerID="a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2"
Oct 30 21:32:23.642164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.642097    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2"} err="failed to get container status \"a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2\": rpc error: code = NotFound desc = could not find container \"a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2\": container with ID starting with a565a5a90f3458deb00a0cb54315140a0e6d32bfed85eaf015f4d5b4b6a089d2 not found: ID does not exist"
Oct 30 21:32:23.652072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.652042    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"]
Oct 30 21:32:23.656756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.656731    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-dpwhn"]
Oct 30 21:32:23.970819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.970790    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:23.971252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.971200    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:23.971888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:23.971866    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:32:24.500982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:24.500949    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 21:32:24.501293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:24.501258    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 21:32:25.081208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:25.081173    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" path="/var/lib/kubelet/pods/e957b4da-6819-42ee-9afa-e89cef9c4bfd/volumes"
Oct 30 21:32:25.240489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:25.240454    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"]
Oct 30 21:32:25.240783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:25.240758    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy" containerID="cri-o://82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528" gracePeriod=30
Oct 30 21:32:25.508178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:25.496026    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 21:32:25.508178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:25.496092    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 21:32:26.490185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:26.490148    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 21:32:26.490775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:26.490217    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 21:32:27.494262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:27.494232    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:32:27.495666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:27.495639    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:32:29.164907 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:32:29.164863    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:32:29.756214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:29.756178    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"]
Oct 30 21:32:29.756449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:29.756420    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" podUID="3b0b01e6-c368-463f-b629-27197fe0239e" containerName="istio-proxy" containerID="cri-o://22e935c97622d08fb778681189e84122b1b4c05b109960b2b422bd5b1b149c54" gracePeriod=30
Oct 30 21:32:30.545042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.544996    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:32:30.584793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.584754    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-socket\") pod \"b0af372e-9d87-4659-ac38-63117bc6321b\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") "
Oct 30 21:32:30.584989 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.584938    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-istio-token\" (UniqueName: \"kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-istio-token\") pod \"b0af372e-9d87-4659-ac38-63117bc6321b\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") "
Oct 30 21:32:30.585069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.585012    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0af372e-9d87-4659-ac38-63117bc6321b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0af372e-9d87-4659-ac38-63117bc6321b-istiod-ca-cert\") pod \"b0af372e-9d87-4659-ac38-63117bc6321b\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") "
Oct 30 21:32:30.585069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.585059    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-envoy\") pod \"b0af372e-9d87-4659-ac38-63117bc6321b\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") "
Oct 30 21:32:30.585194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.585102    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0af372e-9d87-4659-ac38-63117bc6321b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0af372e-9d87-4659-ac38-63117bc6321b-istio-podinfo\") pod \"b0af372e-9d87-4659-ac38-63117bc6321b\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") "
Oct 30 21:32:30.585194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.585177    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-data\") pod \"b0af372e-9d87-4659-ac38-63117bc6321b\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") "
Oct 30 21:32:30.585308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.585217    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-certs\") pod \"b0af372e-9d87-4659-ac38-63117bc6321b\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") "
Oct 30 21:32:30.585868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.585838    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0af372e-9d87-4659-ac38-63117bc6321b-istiod-ca-cert" pod "b0af372e-9d87-4659-ac38-63117bc6321b" (UID: "b0af372e-9d87-4659-ac38-63117bc6321b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:30.587652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.586365    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-certs" pod "b0af372e-9d87-4659-ac38-63117bc6321b" (UID: "b0af372e-9d87-4659-ac38-63117bc6321b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:30.587652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.586476    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-kube-api-access-b2c5q\" (UniqueName: \"kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-kube-api-access-b2c5q\") pod \"b0af372e-9d87-4659-ac38-63117bc6321b\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") "
Oct 30 21:32:30.587652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.586559    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-credential-socket\") pod \"b0af372e-9d87-4659-ac38-63117bc6321b\" (UID: \"b0af372e-9d87-4659-ac38-63117bc6321b\") "
Oct 30 21:32:30.587652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.587321    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.587652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.587343    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0af372e-9d87-4659-ac38-63117bc6321b-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.587652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.587574    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-credential-socket" pod "b0af372e-9d87-4659-ac38-63117bc6321b" (UID: "b0af372e-9d87-4659-ac38-63117bc6321b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:30.588008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.587848    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-data" pod "b0af372e-9d87-4659-ac38-63117bc6321b" (UID: "b0af372e-9d87-4659-ac38-63117bc6321b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:30.588008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.587898    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-socket" pod "b0af372e-9d87-4659-ac38-63117bc6321b" (UID: "b0af372e-9d87-4659-ac38-63117bc6321b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:30.594148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.594120    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-kube-api-access-b2c5q" pod "b0af372e-9d87-4659-ac38-63117bc6321b" (UID: "b0af372e-9d87-4659-ac38-63117bc6321b"). InnerVolumeSpecName "kube-api-access-b2c5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:30.596372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.596339    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0af372e-9d87-4659-ac38-63117bc6321b-istio-podinfo" pod "b0af372e-9d87-4659-ac38-63117bc6321b" (UID: "b0af372e-9d87-4659-ac38-63117bc6321b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:30.602287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.602261    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-envoy" pod "b0af372e-9d87-4659-ac38-63117bc6321b" (UID: "b0af372e-9d87-4659-ac38-63117bc6321b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:30.604097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.604066    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-istio-token" pod "b0af372e-9d87-4659-ac38-63117bc6321b" (UID: "b0af372e-9d87-4659-ac38-63117bc6321b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:30.689525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.689491    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.689525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.689519    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.689525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.689529    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0af372e-9d87-4659-ac38-63117bc6321b-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.689772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.689538    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.689772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.689547    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2c5q\" (UniqueName: \"kubernetes.io/projected/b0af372e-9d87-4659-ac38-63117bc6321b-kube-api-access-b2c5q\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.689772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.689555    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:30.689772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:30.689577    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0af372e-9d87-4659-ac38-63117bc6321b-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:31.185642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.185606    4918 generic.go:356] "Generic (PLEG): container finished" podID="b0af372e-9d87-4659-ac38-63117bc6321b" containerID="82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528" exitCode=0
Oct 30 21:32:31.185839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.185670    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"
Oct 30 21:32:31.185839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.185694    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" event={"ID":"b0af372e-9d87-4659-ac38-63117bc6321b","Type":"ContainerDied","Data":"82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528"}
Oct 30 21:32:31.185839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.185745    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4" event={"ID":"b0af372e-9d87-4659-ac38-63117bc6321b","Type":"ContainerDied","Data":"bdccd95a2f8b70e5e971c4aaa8dda5ed1febc425efa1b23d738502dbe7c19632"}
Oct 30 21:32:31.185839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.185771    4918 scope.go:122] "RemoveContainer" containerID="82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528"
Oct 30 21:32:31.199083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.199060    4918 scope.go:122] "RemoveContainer" containerID="82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528"
Oct 30 21:32:31.199481 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:32:31.199377    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528\": container with ID starting with 82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528 not found: ID does not exist" containerID="82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528"
Oct 30 21:32:31.199560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.199494    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528"} err="failed to get container status \"82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528\": rpc error: code = NotFound desc = could not find container \"82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528\": container with ID starting with 82cb958585ff58bb9a431c1bee84b158bdd492432e004bd0f67e6ec0a3d7c528 not found: ID does not exist"
Oct 30 21:32:31.211327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.211302    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"]
Oct 30 21:32:31.217826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.217795    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-669w4"]
Oct 30 21:32:31.303908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.303870    4918 scope.go:122] "RemoveContainer" containerID="d6d816fcf3521e83131feb379a250e0117a594d9c880a31c7d303856cc02862e"
Oct 30 21:32:31.316256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.316225    4918 scope.go:122] "RemoveContainer" containerID="75393997f786a94d1778f8b9904c10062877e398578df9fcd49bfafc30cf3294"
Oct 30 21:32:31.326151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.326132    4918 scope.go:122] "RemoveContainer" containerID="97ba78aebdb033a1c007f4f26c9ce3b6e75b3789bb31529f6a57eeacf31d79af"
Oct 30 21:32:31.336107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.336090    4918 scope.go:122] "RemoveContainer" containerID="4b9e0c79ef86af5ba7da0c78ca5708de96065dc8e956ab50f909c83f876ff0e7"
Oct 30 21:32:31.345703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.345684    4918 scope.go:122] "RemoveContainer" containerID="91e3812fbd7128599a4be44b8380cc5436fbfc70b537807826f7c12dcf3bc3cf"
Oct 30 21:32:31.355505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.355490    4918 scope.go:122] "RemoveContainer" containerID="a859dbb5e81eb3cb25c5c07a4cd587fbe6b2eb319f7a537d770f379ce6eb4d22"
Oct 30 21:32:31.364837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.364819    4918 scope.go:122] "RemoveContainer" containerID="4014a0830e047858301ca5ffeb6472de7fdd65f6f9f82e90af5b4abc8729b577"
Oct 30 21:32:31.374158 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.374142    4918 scope.go:122] "RemoveContainer" containerID="92c6df04aad3935c76fa48987a898e4400d3ef98a7bc2bfe3592e8f5f6424cfd"
Oct 30 21:32:31.383529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.383512    4918 scope.go:122] "RemoveContainer" containerID="cdaddd6af92fafe3b848b226bd7114cd83ef81249bb1642798390ce829425357"
Oct 30 21:32:31.393084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.393063    4918 scope.go:122] "RemoveContainer" containerID="f63107ee6a91530d607ab6069a19046e44c761364617dc7288356f4443c6815d"
Oct 30 21:32:31.408659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.408565    4918 scope.go:122] "RemoveContainer" containerID="a6da856b6503ec975f18090a67fc73fb3ea26e6588e0572f0cb3e62fb49523db"
Oct 30 21:32:31.420736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.420714    4918 scope.go:122] "RemoveContainer" containerID="52880f9bb0801ba2d6d13acd45a12c46d08f4c543fe391a9ed0b58c5c3686c2a"
Oct 30 21:32:31.431444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.431383    4918 scope.go:122] "RemoveContainer" containerID="8051a32189a028a3ee0b50a1fab9f61db1b9a2a2e6cf581830a59a60d906fb35"
Oct 30 21:32:31.441940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.441920    4918 scope.go:122] "RemoveContainer" containerID="a985bc1b4882ea00ae3dccc336a0c9963732489b73f93ea751ee23ffc8331641"
Oct 30 21:32:31.827939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.827888    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"]
Oct 30 21:32:31.828793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:31.828759    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv" podUID="6d0dc22c-2bed-4082-9a17-c606d796ce30" containerName="istio-proxy" containerID="cri-o://99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379" gracePeriod=30
Oct 30 21:32:33.080380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:33.080344    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" path="/var/lib/kubelet/pods/b0af372e-9d87-4659-ac38-63117bc6321b/volumes"
Oct 30 21:32:35.281274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.281244    4918 generic.go:356] "Generic (PLEG): container finished" podID="3b0b01e6-c368-463f-b629-27197fe0239e" containerID="22e935c97622d08fb778681189e84122b1b4c05b109960b2b422bd5b1b149c54" exitCode=0
Oct 30 21:32:35.281709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.281322    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" event={"ID":"3b0b01e6-c368-463f-b629-27197fe0239e","Type":"ContainerDied","Data":"22e935c97622d08fb778681189e84122b1b4c05b109960b2b422bd5b1b149c54"}
Oct 30 21:32:35.478978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.478951    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:32:35.528630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.528597    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-data\") pod \"3b0b01e6-c368-463f-b629-27197fe0239e\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") "
Oct 30 21:32:35.528796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.528703    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-socket\") pod \"3b0b01e6-c368-463f-b629-27197fe0239e\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") "
Oct 30 21:32:35.528881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.528825    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-kube-api-access-985dv\" (UniqueName: \"kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-kube-api-access-985dv\") pod \"3b0b01e6-c368-463f-b629-27197fe0239e\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") "
Oct 30 21:32:35.528931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.528890    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-data" pod "3b0b01e6-c368-463f-b629-27197fe0239e" (UID: "3b0b01e6-c368-463f-b629-27197fe0239e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:35.528970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.528922    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-socket" pod "3b0b01e6-c368-463f-b629-27197fe0239e" (UID: "3b0b01e6-c368-463f-b629-27197fe0239e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:35.529334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.529313    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-credential-socket\") pod \"3b0b01e6-c368-463f-b629-27197fe0239e\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") "
Oct 30 21:32:35.529447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.529435    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-envoy\") pod \"3b0b01e6-c368-463f-b629-27197fe0239e\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") "
Oct 30 21:32:35.529622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.529554    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-credential-socket" pod "3b0b01e6-c368-463f-b629-27197fe0239e" (UID: "3b0b01e6-c368-463f-b629-27197fe0239e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:35.529622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.529574    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-certs\") pod \"3b0b01e6-c368-463f-b629-27197fe0239e\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") "
Oct 30 21:32:35.529622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.529639    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b0b01e6-c368-463f-b629-27197fe0239e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b01e6-c368-463f-b629-27197fe0239e-istio-podinfo\") pod \"3b0b01e6-c368-463f-b629-27197fe0239e\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") "
Oct 30 21:32:35.529869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.529675    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0b01e6-c368-463f-b629-27197fe0239e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b01e6-c368-463f-b629-27197fe0239e-istiod-ca-cert\") pod \"3b0b01e6-c368-463f-b629-27197fe0239e\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") "
Oct 30 21:32:35.529869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.529761    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-istio-token\") pod \"3b0b01e6-c368-463f-b629-27197fe0239e\" (UID: \"3b0b01e6-c368-463f-b629-27197fe0239e\") "
Oct 30 21:32:35.530504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.530447    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0b01e6-c368-463f-b629-27197fe0239e-istiod-ca-cert" pod "3b0b01e6-c368-463f-b629-27197fe0239e" (UID: "3b0b01e6-c368-463f-b629-27197fe0239e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:35.530946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.530922    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:35.531077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.530950    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b0b01e6-c368-463f-b629-27197fe0239e-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:35.531077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.530965    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:35.531077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.530981    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:35.531077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.531049    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-certs" pod "3b0b01e6-c368-463f-b629-27197fe0239e" (UID: "3b0b01e6-c368-463f-b629-27197fe0239e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:35.532427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.532342    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-kube-api-access-985dv" pod "3b0b01e6-c368-463f-b629-27197fe0239e" (UID: "3b0b01e6-c368-463f-b629-27197fe0239e"). InnerVolumeSpecName "kube-api-access-985dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:35.532533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.532431    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b0b01e6-c368-463f-b629-27197fe0239e-istio-podinfo" pod "3b0b01e6-c368-463f-b629-27197fe0239e" (UID: "3b0b01e6-c368-463f-b629-27197fe0239e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:35.532650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.532631    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-envoy" pod "3b0b01e6-c368-463f-b629-27197fe0239e" (UID: "3b0b01e6-c368-463f-b629-27197fe0239e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:35.533381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.533355    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-istio-token" pod "3b0b01e6-c368-463f-b629-27197fe0239e" (UID: "3b0b01e6-c368-463f-b629-27197fe0239e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:35.634471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.633375    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-985dv\" (UniqueName: \"kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-kube-api-access-985dv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:35.634471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.633431    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:35.634471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.633444    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b0b01e6-c368-463f-b629-27197fe0239e-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:35.634471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.633459    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b0b01e6-c368-463f-b629-27197fe0239e-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:35.634471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.633468    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b0b01e6-c368-463f-b629-27197fe0239e-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:35.809659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.809570    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"]
Oct 30 21:32:35.811324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.811300    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" containerName="istio-proxy"
Oct 30 21:32:35.811324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.811323    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" containerName="istio-proxy"
Oct 30 21:32:35.811572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.811410    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy"
Oct 30 21:32:35.811572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.811418    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy"
Oct 30 21:32:35.811572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.811450    4918 cpu_manager.go:414] "containerMap: removing container" podUID="0ebbec47-f69f-4f8a-997f-4b1be1120e4c" containerName="revoke-keys"
Oct 30 21:32:35.811572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.811458    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebbec47-f69f-4f8a-997f-4b1be1120e4c" containerName="revoke-keys"
Oct 30 21:32:35.811572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.811473    4918 cpu_manager.go:414] "containerMap: removing container" podUID="3b0b01e6-c368-463f-b629-27197fe0239e" containerName="istio-proxy"
Oct 30 21:32:35.811572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.811481    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0b01e6-c368-463f-b629-27197fe0239e" containerName="istio-proxy"
Oct 30 21:32:35.811952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.811934    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e957b4da-6819-42ee-9afa-e89cef9c4bfd" containerName="istio-proxy"
Oct 30 21:32:35.812018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.811970    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0af372e-9d87-4659-ac38-63117bc6321b" containerName="istio-proxy"
Oct 30 21:32:35.812018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.811991    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebbec47-f69f-4f8a-997f-4b1be1120e4c" containerName="revoke-keys"
Oct 30 21:32:35.812018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.812004    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0b01e6-c368-463f-b629-27197fe0239e" containerName="istio-proxy"
Oct 30 21:32:35.817908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.817889    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"
Oct 30 21:32:35.830865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.829725    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"]
Oct 30 21:32:35.884327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.884295    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7b5a7c-dockercfg-4qt7n\""
Oct 30 21:32:35.884527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.884332    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7b5a7c\""
Oct 30 21:32:35.942669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.942640    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnm29\" (UniqueName: \"kubernetes.io/projected/18762808-b2c0-4c82-bda9-4b89c4a882c5-kube-api-access-xnm29\") pod \"payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4\" (UID: \"18762808-b2c0-4c82-bda9-4b89c4a882c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"
Oct 30 21:32:35.942847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:35.942815    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18762808-b2c0-4c82-bda9-4b89c4a882c5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4\" (UID: \"18762808-b2c0-4c82-bda9-4b89c4a882c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"
Oct 30 21:32:36.063611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:36.062896    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18762808-b2c0-4c82-bda9-4b89c4a882c5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4\" (UID: \"18762808-b2c0-4c82-bda9-4b89c4a882c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"
Oct 30 21:32:36.063611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:36.063163    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnm29\" (UniqueName: \"kubernetes.io/projected/18762808-b2c0-4c82-bda9-4b89c4a882c5-kube-api-access-xnm29\") pod \"payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4\" (UID: \"18762808-b2c0-4c82-bda9-4b89c4a882c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"
Oct 30 21:32:36.114076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:36.106131    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18762808-b2c0-4c82-bda9-4b89c4a882c5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4\" (UID: \"18762808-b2c0-4c82-bda9-4b89c4a882c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"
Oct 30 21:32:36.117412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:36.117088    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnm29\" (UniqueName: \"kubernetes.io/projected/18762808-b2c0-4c82-bda9-4b89c4a882c5-kube-api-access-xnm29\") pod \"payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4\" (UID: \"18762808-b2c0-4c82-bda9-4b89c4a882c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"
Oct 30 21:32:36.146215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:36.145709    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"
Oct 30 21:32:36.349498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:36.346688    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj" event={"ID":"3b0b01e6-c368-463f-b629-27197fe0239e","Type":"ContainerDied","Data":"5e1559b62fa3d1164f4cadbbee0b82e0cbfde07fc5ac7512a0100cdb4446f519"}
Oct 30 21:32:36.349498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:36.346798    4918 scope.go:122] "RemoveContainer" containerID="22e935c97622d08fb778681189e84122b1b4c05b109960b2b422bd5b1b149c54"
Oct 30 21:32:36.349498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:36.347047    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"
Oct 30 21:32:36.392045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:36.391964    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"]
Oct 30 21:32:36.405267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:36.405173    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-5mkwj"]
Oct 30 21:32:36.667762 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:32:36.667721    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18762808_b2c0_4c82_bda9_4b89c4a882c5.slice/crio-24d990f163b3cd6331259010d908711a24e87c3b8bb602d44d2c9fb067ceaadc WatchSource:0}: Error finding container 24d990f163b3cd6331259010d908711a24e87c3b8bb602d44d2c9fb067ceaadc: Status 404 returned error can't find the container with id 24d990f163b3cd6331259010d908711a24e87c3b8bb602d44d2c9fb067ceaadc
Oct 30 21:32:36.668528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:36.668508    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"]
Oct 30 21:32:37.127887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.127854    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0b01e6-c368-463f-b629-27197fe0239e" path="/var/lib/kubelet/pods/3b0b01e6-c368-463f-b629-27197fe0239e/volumes"
Oct 30 21:32:37.211216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.211150    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:32:37.324983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.324950    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-data\") pod \"6d0dc22c-2bed-4082-9a17-c606d796ce30\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") "
Oct 30 21:32:37.325230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325011    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0dc22c-2bed-4082-9a17-c606d796ce30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0dc22c-2bed-4082-9a17-c606d796ce30-istiod-ca-cert\") pod \"6d0dc22c-2bed-4082-9a17-c606d796ce30\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") "
Oct 30 21:32:37.325230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325071    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-podinfo\") pod \"6d0dc22c-2bed-4082-9a17-c606d796ce30\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") "
Oct 30 21:32:37.325230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325182    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-certs\") pod \"6d0dc22c-2bed-4082-9a17-c606d796ce30\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") "
Oct 30 21:32:37.325427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325256    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-kube-api-access-5v7j5\" (UniqueName: \"kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-kube-api-access-5v7j5\") pod \"6d0dc22c-2bed-4082-9a17-c606d796ce30\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") "
Oct 30 21:32:37.325427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325261    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-data" pod "6d0dc22c-2bed-4082-9a17-c606d796ce30" (UID: "6d0dc22c-2bed-4082-9a17-c606d796ce30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:37.325427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325348    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-credential-socket\") pod \"6d0dc22c-2bed-4082-9a17-c606d796ce30\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") "
Oct 30 21:32:37.325593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325493    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-socket\") pod \"6d0dc22c-2bed-4082-9a17-c606d796ce30\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") "
Oct 30 21:32:37.325593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325527    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-certs" pod "6d0dc22c-2bed-4082-9a17-c606d796ce30" (UID: "6d0dc22c-2bed-4082-9a17-c606d796ce30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:37.325593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325583    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-envoy\") pod \"6d0dc22c-2bed-4082-9a17-c606d796ce30\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") "
Oct 30 21:32:37.325743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325603    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-credential-socket" pod "6d0dc22c-2bed-4082-9a17-c606d796ce30" (UID: "6d0dc22c-2bed-4082-9a17-c606d796ce30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:37.325743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325625    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0dc22c-2bed-4082-9a17-c606d796ce30-istiod-ca-cert" pod "6d0dc22c-2bed-4082-9a17-c606d796ce30" (UID: "6d0dc22c-2bed-4082-9a17-c606d796ce30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:37.325743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325662    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-token\" (UniqueName: \"kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-token\") pod \"6d0dc22c-2bed-4082-9a17-c606d796ce30\" (UID: \"6d0dc22c-2bed-4082-9a17-c606d796ce30\") "
Oct 30 21:32:37.325895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.325752    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-socket" pod "6d0dc22c-2bed-4082-9a17-c606d796ce30" (UID: "6d0dc22c-2bed-4082-9a17-c606d796ce30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:37.327892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.327864    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-kube-api-access-5v7j5" pod "6d0dc22c-2bed-4082-9a17-c606d796ce30" (UID: "6d0dc22c-2bed-4082-9a17-c606d796ce30"). InnerVolumeSpecName "kube-api-access-5v7j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:37.328213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.328189    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-token" pod "6d0dc22c-2bed-4082-9a17-c606d796ce30" (UID: "6d0dc22c-2bed-4082-9a17-c606d796ce30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:37.328339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.328238    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.328339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.328256    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.328339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.328272    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0dc22c-2bed-4082-9a17-c606d796ce30-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.328339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.328282    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.328339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.328290    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v7j5\" (UniqueName: \"kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-kube-api-access-5v7j5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.328339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.328299    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.328596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.328425    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-envoy" pod "6d0dc22c-2bed-4082-9a17-c606d796ce30" (UID: "6d0dc22c-2bed-4082-9a17-c606d796ce30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:37.328596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.328438    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-podinfo" pod "6d0dc22c-2bed-4082-9a17-c606d796ce30" (UID: "6d0dc22c-2bed-4082-9a17-c606d796ce30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:37.356909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.356883    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4" event={"ID":"18762808-b2c0-4c82-bda9-4b89c4a882c5","Type":"ContainerStarted","Data":"661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4"}
Oct 30 21:32:37.357341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.356938    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4" event={"ID":"18762808-b2c0-4c82-bda9-4b89c4a882c5","Type":"ContainerStarted","Data":"24d990f163b3cd6331259010d908711a24e87c3b8bb602d44d2c9fb067ceaadc"}
Oct 30 21:32:37.357341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.356979    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"
Oct 30 21:32:37.359817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.359792    4918 generic.go:356] "Generic (PLEG): container finished" podID="6d0dc22c-2bed-4082-9a17-c606d796ce30" containerID="99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379" exitCode=0
Oct 30 21:32:37.359961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.359869    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"
Oct 30 21:32:37.359961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.359881    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv" event={"ID":"6d0dc22c-2bed-4082-9a17-c606d796ce30","Type":"ContainerDied","Data":"99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379"}
Oct 30 21:32:37.359961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.359930    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv" event={"ID":"6d0dc22c-2bed-4082-9a17-c606d796ce30","Type":"ContainerDied","Data":"0284556302ed4d3fb59f8fa4f7450862969cb5055f94916b3b4f850c6dd78c5c"}
Oct 30 21:32:37.359961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.359958    4918 scope.go:122] "RemoveContainer" containerID="99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379"
Oct 30 21:32:37.388487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.388458    4918 scope.go:122] "RemoveContainer" containerID="99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379"
Oct 30 21:32:37.389087 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:32:37.389055    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379\": container with ID starting with 99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379 not found: ID does not exist" containerID="99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379"
Oct 30 21:32:37.389218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.389100    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379"} err="failed to get container status \"99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379\": rpc error: code = NotFound desc = could not find container \"99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379\": container with ID starting with 99d717664cc501e39f482e872bea45c57ea3b4749df40a5e1eeb87df26688379 not found: ID does not exist"
Oct 30 21:32:37.390749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.390698    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4" podStartSLOduration=2.390685371 podStartE2EDuration="2.390685371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:37.387246693 +0000 UTC m=+1870.433805839" watchObservedRunningTime="2026-10-30 21:32:37.390685371 +0000 UTC m=+1870.437244515"
Oct 30 21:32:37.413925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.413889    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"]
Oct 30 21:32:37.419689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.419653    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-6nhzv"]
Oct 30 21:32:37.430228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.430199    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.430228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.430230    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:37.430463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:37.430245    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0dc22c-2bed-4082-9a17-c606d796ce30-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:39.081921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:39.081892    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0dc22c-2bed-4082-9a17-c606d796ce30" path="/var/lib/kubelet/pods/6d0dc22c-2bed-4082-9a17-c606d796ce30/volumes"
Oct 30 21:32:41.344445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.344409    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"]
Oct 30 21:32:41.345847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.345821    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6d0dc22c-2bed-4082-9a17-c606d796ce30" containerName="istio-proxy"
Oct 30 21:32:41.345847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.345847    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0dc22c-2bed-4082-9a17-c606d796ce30" containerName="istio-proxy"
Oct 30 21:32:41.346033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.346021    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0dc22c-2bed-4082-9a17-c606d796ce30" containerName="istio-proxy"
Oct 30 21:32:41.352703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.352680    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"]
Oct 30 21:32:41.352836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.352788    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"
Oct 30 21:32:41.358029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.358004    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0ea1ea-dockercfg-nmlc4\""
Oct 30 21:32:41.413676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.408898    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0ea1ea\""
Oct 30 21:32:41.413676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.411448    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836991d9-b095-44e2-83af-b8a260eb2413-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb\" (UID: \"836991d9-b095-44e2-83af-b8a260eb2413\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"
Oct 30 21:32:41.413676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.411704    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swppg\" (UniqueName: \"kubernetes.io/projected/836991d9-b095-44e2-83af-b8a260eb2413-kube-api-access-swppg\") pod \"payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb\" (UID: \"836991d9-b095-44e2-83af-b8a260eb2413\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"
Oct 30 21:32:41.539413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.535902    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swppg\" (UniqueName: \"kubernetes.io/projected/836991d9-b095-44e2-83af-b8a260eb2413-kube-api-access-swppg\") pod \"payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb\" (UID: \"836991d9-b095-44e2-83af-b8a260eb2413\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"
Oct 30 21:32:41.539413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.536070    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836991d9-b095-44e2-83af-b8a260eb2413-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb\" (UID: \"836991d9-b095-44e2-83af-b8a260eb2413\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"
Oct 30 21:32:41.636739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.635693    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836991d9-b095-44e2-83af-b8a260eb2413-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb\" (UID: \"836991d9-b095-44e2-83af-b8a260eb2413\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"
Oct 30 21:32:41.718510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.712134    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swppg\" (UniqueName: \"kubernetes.io/projected/836991d9-b095-44e2-83af-b8a260eb2413-kube-api-access-swppg\") pod \"payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb\" (UID: \"836991d9-b095-44e2-83af-b8a260eb2413\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"
Oct 30 21:32:41.976554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:41.972702    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"
Oct 30 21:32:42.313596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:42.307094    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"]
Oct 30 21:32:42.315164 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:32:42.315132    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836991d9_b095_44e2_83af_b8a260eb2413.slice/crio-9781e04445ae6fdc1bf5ab18173d6fa52578166e2c7c74907dfdd1431ca19bb7 WatchSource:0}: Error finding container 9781e04445ae6fdc1bf5ab18173d6fa52578166e2c7c74907dfdd1431ca19bb7: Status 404 returned error can't find the container with id 9781e04445ae6fdc1bf5ab18173d6fa52578166e2c7c74907dfdd1431ca19bb7
Oct 30 21:32:42.722473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:42.707492    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb" event={"ID":"836991d9-b095-44e2-83af-b8a260eb2413","Type":"ContainerStarted","Data":"ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650"}
Oct 30 21:32:42.722473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:42.707566    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb" event={"ID":"836991d9-b095-44e2-83af-b8a260eb2413","Type":"ContainerStarted","Data":"9781e04445ae6fdc1bf5ab18173d6fa52578166e2c7c74907dfdd1431ca19bb7"}
Oct 30 21:32:42.722473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:42.708902    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"
Oct 30 21:32:42.784454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:42.774591    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb" podStartSLOduration=1.774568113 podStartE2EDuration="1.774568113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:42.744925041 +0000 UTC m=+1875.791484188" watchObservedRunningTime="2026-10-30 21:32:42.774568113 +0000 UTC m=+1875.821127259"
Oct 30 21:32:43.702418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:43.692255    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:43.919416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:43.896883    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:43.936766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:43.935718    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:32:43.965041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:43.963693    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:32:43.966619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:43.966583    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:44.084181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:44.083534    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:44.148573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:44.147510    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:32:48.411363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:48.404797    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"
Oct 30 21:32:48.569264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:48.569236    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"]
Oct 30 21:32:48.569488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:48.569463    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw" podUID="1c783f14-4102-4a9a-a62d-4c971d7dd31b" containerName="istio-proxy" containerID="cri-o://e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5" gracePeriod=30
Oct 30 21:32:54.248645 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.248618    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:32:54.346198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.346115    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-certs\") pod \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") "
Oct 30 21:32:54.346198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.346153    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-envoy\") pod \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") "
Oct 30 21:32:54.346198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.346173    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-socket\") pod \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") "
Oct 30 21:32:54.346547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.346270    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-token\" (UniqueName: \"kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-token\") pod \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") "
Oct 30 21:32:54.346547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.346305    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-kube-api-access-hjjmr\" (UniqueName: \"kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-kube-api-access-hjjmr\") pod \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") "
Oct 30 21:32:54.346547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.346467    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-credential-socket\") pod \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") "
Oct 30 21:32:54.346547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.346508    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istiod-ca-cert\") pod \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") "
Oct 30 21:32:54.347805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.346564    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-podinfo\") pod \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") "
Oct 30 21:32:54.347805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.346630    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-data\") pod \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\" (UID: \"1c783f14-4102-4a9a-a62d-4c971d7dd31b\") "
Oct 30 21:32:54.347805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.346774    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-certs" pod "1c783f14-4102-4a9a-a62d-4c971d7dd31b" (UID: "1c783f14-4102-4a9a-a62d-4c971d7dd31b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:54.347805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.347712    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istiod-ca-cert" pod "1c783f14-4102-4a9a-a62d-4c971d7dd31b" (UID: "1c783f14-4102-4a9a-a62d-4c971d7dd31b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:32:54.349024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.348848    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:54.349024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.348875    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:54.350304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.350267    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-socket" pod "1c783f14-4102-4a9a-a62d-4c971d7dd31b" (UID: "1c783f14-4102-4a9a-a62d-4c971d7dd31b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:54.350383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.350329    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-data" pod "1c783f14-4102-4a9a-a62d-4c971d7dd31b" (UID: "1c783f14-4102-4a9a-a62d-4c971d7dd31b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:54.350556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.350518    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-credential-socket" pod "1c783f14-4102-4a9a-a62d-4c971d7dd31b" (UID: "1c783f14-4102-4a9a-a62d-4c971d7dd31b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:54.352863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.352832    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-envoy" pod "1c783f14-4102-4a9a-a62d-4c971d7dd31b" (UID: "1c783f14-4102-4a9a-a62d-4c971d7dd31b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:32:54.352979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.352920    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-podinfo" pod "1c783f14-4102-4a9a-a62d-4c971d7dd31b" (UID: "1c783f14-4102-4a9a-a62d-4c971d7dd31b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:32:54.353484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.353456    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-token" pod "1c783f14-4102-4a9a-a62d-4c971d7dd31b" (UID: "1c783f14-4102-4a9a-a62d-4c971d7dd31b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:54.357262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.357236    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-kube-api-access-hjjmr" pod "1c783f14-4102-4a9a-a62d-4c971d7dd31b" (UID: "1c783f14-4102-4a9a-a62d-4c971d7dd31b"). InnerVolumeSpecName "kube-api-access-hjjmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:32:54.450950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.450916    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:54.450950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.450945    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:54.450950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.450954    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:54.451284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.450963    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:54.451284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.450974    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c783f14-4102-4a9a-a62d-4c971d7dd31b-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:54.451284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.450982    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:54.451284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.450992    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjjmr\" (UniqueName: \"kubernetes.io/projected/1c783f14-4102-4a9a-a62d-4c971d7dd31b-kube-api-access-hjjmr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:32:54.857697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:54.857670    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"
Oct 30 21:32:55.046542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:55.046508    4918 generic.go:356] "Generic (PLEG): container finished" podID="1c783f14-4102-4a9a-a62d-4c971d7dd31b" containerID="e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5" exitCode=0
Oct 30 21:32:55.046752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:55.046594    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"
Oct 30 21:32:55.046752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:55.046590    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw" event={"ID":"1c783f14-4102-4a9a-a62d-4c971d7dd31b","Type":"ContainerDied","Data":"e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5"}
Oct 30 21:32:55.046752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:55.046749    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw" event={"ID":"1c783f14-4102-4a9a-a62d-4c971d7dd31b","Type":"ContainerDied","Data":"ebf576c86629fac8cec113307ef5bad79b93681e665ca7725320139bb8817289"}
Oct 30 21:32:55.046926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:55.046773    4918 scope.go:122] "RemoveContainer" containerID="e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5"
Oct 30 21:32:55.058932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:55.058654    4918 scope.go:122] "RemoveContainer" containerID="e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5"
Oct 30 21:32:55.059292 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:32:55.059246    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5\": container with ID starting with e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5 not found: ID does not exist" containerID="e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5"
Oct 30 21:32:55.059442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:55.059303    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5"} err="failed to get container status \"e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5\": rpc error: code = NotFound desc = could not find container \"e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5\": container with ID starting with e44f0a9a5b7ac79372af78a93faba9a201c617f63a45ff29a9f8c8da9ace21c5 not found: ID does not exist"
Oct 30 21:32:55.087498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:55.087468    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"]
Oct 30 21:32:55.097894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:55.097865    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qm8sw"]
Oct 30 21:32:56.058472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:56.058438    4918 generic.go:356] "Generic (PLEG): container finished" podID="f7574078-8dcf-4713-9574-c489ef73a943" containerID="0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934" exitCode=0
Oct 30 21:32:56.058941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:56.058508    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" event={"ID":"f7574078-8dcf-4713-9574-c489ef73a943","Type":"ContainerDied","Data":"0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934"}
Oct 30 21:32:56.058941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:56.058926    4918 scope.go:122] "RemoveContainer" containerID="0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934"
Oct 30 21:32:56.060730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:56.060703    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:32:56.060850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:56.060770    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:32:56.060850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:56.060815    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:32:57.079128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:57.079094    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c783f14-4102-4a9a-a62d-4c971d7dd31b" path="/var/lib/kubelet/pods/1c783f14-4102-4a9a-a62d-4c971d7dd31b/volumes"
Oct 30 21:32:57.079554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:57.079468    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" event={"ID":"f7574078-8dcf-4713-9574-c489ef73a943","Type":"ContainerStarted","Data":"4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827"}
Oct 30 21:32:57.149931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:57.149900    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:32:57.150143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:57.150119    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 21:32:57.150229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:57.150179    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 21:32:58.150683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:58.150651    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 21:32:58.151095 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:58.150707    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 21:32:59.150426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:59.150377    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 21:32:59.150599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:32:59.150452    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 21:32:59.162677 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:32:59.162643    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:33:00.150652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:00.150618    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 21:33:00.150855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:00.150674    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 21:33:01.150037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:01.150002    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 21:33:01.150501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:01.150064    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 21:33:02.150666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:02.150632    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body=
Oct 30 21:33:02.151173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:02.150693    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused"
Oct 30 21:33:02.841993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:02.841964    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:33:03.154981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:03.154905    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:33:03.156266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:03.156241    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:33:18.412858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.412815    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"]
Oct 30 21:33:18.414006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.413986    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1c783f14-4102-4a9a-a62d-4c971d7dd31b" containerName="istio-proxy"
Oct 30 21:33:18.414006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.414007    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c783f14-4102-4a9a-a62d-4c971d7dd31b" containerName="istio-proxy"
Oct 30 21:33:18.414166 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.414157    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c783f14-4102-4a9a-a62d-4c971d7dd31b" containerName="istio-proxy"
Oct 30 21:33:18.418070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.418052    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.422976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.422947    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b5a7c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-7b5a7c\""
Oct 30 21:33:18.424698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.424679    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b5a7c\"/\"kube-root-ca.crt\""
Oct 30 21:33:18.424991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.424914    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b5a7c\"/\"openshift-service-ca.crt\""
Oct 30 21:33:18.425101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.425002    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b5a7c\"/\"payload-processing-external-model-e2e--de79f2c3-dockercfg-8d5x7\""
Oct 30 21:33:18.425101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.425039    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-7b5a7c\"/\"agc-cred-21e07f9bc70b\""
Oct 30 21:33:18.425101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.425095    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-7b5a7c\"/\"routing-overlay\""
Oct 30 21:33:18.432014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.431991    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"]
Oct 30 21:33:18.604061 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.604018    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") " pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.605974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.605943    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") " pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.606338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.606311    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") " pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.606573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.606555    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bh6l\" (UniqueName: \"kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-kube-api-access-5bh6l\") pod \"payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") " pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.711464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.710475    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") " pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.711464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.710590    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bh6l\" (UniqueName: \"kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-kube-api-access-5bh6l\") pod \"payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") " pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.711464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.710766    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") " pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.711464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.710824    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") " pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.711869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.711740    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") " pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.716414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.713189    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") " pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.719498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.719474    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") " pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.735426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.735138    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bh6l\" (UniqueName: \"kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-kube-api-access-5bh6l\") pod \"payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") " pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:18.754445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:18.750664    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:19.149378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:19.149348    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"]
Oct 30 21:33:19.150524 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:33:19.150492    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d4dd38_bf08_4b1d_bd2a_ece135534a55.slice/crio-91c918b36ad4399c0ef273feff76161b1586fced34b6910491cb2dc005a41c19 WatchSource:0}: Error finding container 91c918b36ad4399c0ef273feff76161b1586fced34b6910491cb2dc005a41c19: Status 404 returned error can't find the container with id 91c918b36ad4399c0ef273feff76161b1586fced34b6910491cb2dc005a41c19
Oct 30 21:33:19.444810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:19.444707    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj" event={"ID":"26d4dd38-bf08-4b1d-bd2a-ece135534a55","Type":"ContainerStarted","Data":"e068ea1becb5a9ebb8bc739e53f73c7a6de4df42deb7ab6d969ff15de0428ef2"}
Oct 30 21:33:19.444810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:19.444771    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj" event={"ID":"26d4dd38-bf08-4b1d-bd2a-ece135534a55","Type":"ContainerStarted","Data":"91c918b36ad4399c0ef273feff76161b1586fced34b6910491cb2dc005a41c19"}
Oct 30 21:33:19.444810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:19.444801    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:19.466079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:19.466022    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj" podStartSLOduration=1.4660080770000001 podStartE2EDuration="1.466008077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:19.462923174 +0000 UTC m=+1912.509482319" watchObservedRunningTime="2026-10-30 21:33:19.466008077 +0000 UTC m=+1912.512567244"
Oct 30 21:33:22.493375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.493344    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"]
Oct 30 21:33:22.498340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.498320    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:22.504106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.504083    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:33:22.504292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.504086    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:33:22.504496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.504165    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:33:22.504621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.504168    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:33:22.506699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.506680    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"]
Oct 30 21:33:22.656172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.656135    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26v7d\" (UniqueName: \"kubernetes.io/projected/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-kube-api-access-26v7d\") pod \"maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp\" (UID: \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:22.656412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.656330    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp\" (UID: \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:22.656645 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.656624    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp\" (UID: \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:22.758147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.758058    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26v7d\" (UniqueName: \"kubernetes.io/projected/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-kube-api-access-26v7d\") pod \"maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp\" (UID: \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:22.758341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.758153    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp\" (UID: \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:22.758341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.758313    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp\" (UID: \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:22.758826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.758805    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp\" (UID: \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:22.760871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.760846    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp\" (UID: \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:22.768403 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.768369    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26v7d\" (UniqueName: \"kubernetes.io/projected/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-kube-api-access-26v7d\") pod \"maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp\" (UID: \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:22.816366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:22.816339    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:23.097112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:23.097087    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"]
Oct 30 21:33:23.097514 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:33:23.097492    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaead39d9_7bdd_4f97_a9c7_239c0fdbd1a3.slice/crio-d08805c94a4f6aaf0edec4e76ad19e97b69b966e61b6a9a89184de6207825907 WatchSource:0}: Error finding container d08805c94a4f6aaf0edec4e76ad19e97b69b966e61b6a9a89184de6207825907: Status 404 returned error can't find the container with id d08805c94a4f6aaf0edec4e76ad19e97b69b966e61b6a9a89184de6207825907
Oct 30 21:33:23.489047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:23.489012    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp" event={"ID":"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3","Type":"ContainerStarted","Data":"d08805c94a4f6aaf0edec4e76ad19e97b69b966e61b6a9a89184de6207825907"}
Oct 30 21:33:24.500588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:24.500556    4918 generic.go:356] "Generic (PLEG): container finished" podID="aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3" containerID="3cc91604cd09d842359ea900e2e953f510bdaa1dfa42fede700bed538b25994a" exitCode=0
Oct 30 21:33:24.501114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:24.500650    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp" event={"ID":"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3","Type":"ContainerDied","Data":"3cc91604cd09d842359ea900e2e953f510bdaa1dfa42fede700bed538b25994a"}
Oct 30 21:33:25.783652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:25.783592    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:25.948010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:25.947982    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-kube-api-access-26v7d\" (UniqueName: \"kubernetes.io/projected/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-kube-api-access-26v7d\") pod \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\" (UID: \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\") "
Oct 30 21:33:25.948198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:25.948147    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-xks-ca\") pod \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\" (UID: \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\") "
Oct 30 21:33:25.948246 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:25.948194    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-openshift-ca\") pod \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\" (UID: \"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3\") "
Oct 30 21:33:25.948701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:25.948623    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-openshift-ca" pod "aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3" (UID: "aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:33:25.949037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:25.949018    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:25.950377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:25.950346    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-kube-api-access-26v7d" pod "aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3" (UID: "aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3"). InnerVolumeSpecName "kube-api-access-26v7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:33:25.950566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:25.950548    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-xks-ca" pod "aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3" (UID: "aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:33:26.051135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:26.051061    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26v7d\" (UniqueName: \"kubernetes.io/projected/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-kube-api-access-26v7d\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:26.051135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:26.051089    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:33:26.517953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:26.517920    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp" event={"ID":"aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3","Type":"ContainerDied","Data":"d08805c94a4f6aaf0edec4e76ad19e97b69b966e61b6a9a89184de6207825907"}
Oct 30 21:33:26.517953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:26.517958    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"
Oct 30 21:33:26.518186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:26.517969    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d08805c94a4f6aaf0edec4e76ad19e97b69b966e61b6a9a89184de6207825907"
Oct 30 21:33:29.447417 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:33:29.447364    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:33:30.455988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:30.455954    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:33:31.598822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:31.598791    4918 scope.go:122] "RemoveContainer" containerID="ec1006a65e7cc919823c51aefe2ed3bc0819b451a19e927537c8454ef098a0f9"
Oct 30 21:33:31.610700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:31.610678    4918 scope.go:122] "RemoveContainer" containerID="b9cff1ea8970f951fca7eba75cb519eebe0b1e723103dffd9a92eb3d4960cd50"
Oct 30 21:33:35.014767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.012360    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"]
Oct 30 21:33:35.014767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.013759    4918 cpu_manager.go:414] "containerMap: removing container" podUID="aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3" containerName="revoke-keys"
Oct 30 21:33:35.014767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.013782    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3" containerName="revoke-keys"
Oct 30 21:33:35.014767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.013986    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3" containerName="revoke-keys"
Oct 30 21:33:35.041471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.037443    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.060108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.055809    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"]
Oct 30 21:33:35.082409 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.069460    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.099739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.099708    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"]
Oct 30 21:33:35.107498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.107468    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"]
Oct 30 21:33:35.121787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.121764    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-02ecd7-openshift-default-dockercfg-drrk7\""
Oct 30 21:33:35.139697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.139674    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-data\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.139844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.139829    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-credential-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.139904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.139855    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istiod-ca-cert\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.139973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.139947    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbm5b\" (UniqueName: \"kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-kube-api-access-qbm5b\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.140027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.139995    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-podinfo\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.140068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140042    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec92974-c36a-4577-b40d-685b6d039c7d-istiod-ca-cert\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.140068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140058    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-token\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.140185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140088    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-token\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.140185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140160    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbflk\" (UniqueName: \"kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-kube-api-access-gbflk\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.140321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140303    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-podinfo\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.140383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140354    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-credential-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.140533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140514    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-envoy\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.140577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140547    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-certs\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.140668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140652    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-envoy\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.140715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140684    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-data\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.140767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140713    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.140817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140766    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-certs\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.140898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.140876    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.242593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.242564    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-credential-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.242812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.242598    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istiod-ca-cert\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.242812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.242712    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbm5b\" (UniqueName: \"kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-kube-api-access-qbm5b\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.242812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.242754    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-podinfo\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.243014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.242822    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec92974-c36a-4577-b40d-685b6d039c7d-istiod-ca-cert\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.243014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.242847    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-token\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.243014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.242915    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-token\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.243014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.242944    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbflk\" (UniqueName: \"kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-kube-api-access-gbflk\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.243243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243028    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-podinfo\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.243243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243069    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-credential-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.243243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243076    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-credential-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.243438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243331    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-envoy\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.243438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243360    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-certs\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.243438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243368    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istiod-ca-cert\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.243625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243582    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec92974-c36a-4577-b40d-685b6d039c7d-istiod-ca-cert\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.243688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243620    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-credential-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.243688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243637    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-envoy\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.243688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243672    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-data\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.243869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243699    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.243869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243744    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-certs\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.243869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243805    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.243869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243857    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-data\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.244097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.243899    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-certs\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.244292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.244159    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-data\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.244463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.244442    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-certs\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.244644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.244621    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.244734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.244648    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-socket\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.244890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.244866    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-data\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.246117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.246096    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-envoy\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.246424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.246367    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-podinfo\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.246424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.246405    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-podinfo\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.246931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.246910    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-envoy\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.255452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.255359    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-token\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.255647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.255626    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbm5b\" (UniqueName: \"kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-kube-api-access-qbm5b\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.256073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.256030    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-token\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.256238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.256220    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbflk\" (UniqueName: \"kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-kube-api-access-gbflk\") pod \"e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") " pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.370208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.370124    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:35.403123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.403095    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:35.666935 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:33:35.666906    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec92974_c36a_4577_b40d_685b6d039c7d.slice/crio-c054804a11f1193418a36adc1b8e34f0ed62a864aae82a263cd76de63b78b3dd WatchSource:0}: Error finding container c054804a11f1193418a36adc1b8e34f0ed62a864aae82a263cd76de63b78b3dd: Status 404 returned error can't find the container with id c054804a11f1193418a36adc1b8e34f0ed62a864aae82a263cd76de63b78b3dd
Oct 30 21:33:35.667216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.667193    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"]
Oct 30 21:33:35.670535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.670502    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:33:35.670621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.670569    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:33:35.670621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.670599    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:33:35.710247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.710223    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"]
Oct 30 21:33:35.711873 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:33:35.711842    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afa42af_0e54_4c13_80fa_e0d66c519b4a.slice/crio-c8dc99f788e0eea1c1af78df0ae9c3c302edb65a21bbaf0d5c536674a82210bf WatchSource:0}: Error finding container c8dc99f788e0eea1c1af78df0ae9c3c302edb65a21bbaf0d5c536674a82210bf: Status 404 returned error can't find the container with id c8dc99f788e0eea1c1af78df0ae9c3c302edb65a21bbaf0d5c536674a82210bf
Oct 30 21:33:35.714539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.714506    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:33:35.714648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.714584    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:33:35.714648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:35.714629    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:33:36.023251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.023222    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"]
Oct 30 21:33:36.028157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.028135    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.056452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.055839    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0ea1ea-openshift-default-dockercfg-9tksn\""
Oct 30 21:33:36.063481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.063452    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"]
Oct 30 21:33:36.161155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.161123    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-credential-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.161329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.161162    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-istio-token\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.161329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.161196    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389006f4-6c3d-4519-8744-46882675e05b-istiod-ca-cert\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.161329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.161303    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.161509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.161380    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-certs\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.161509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.161465    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-data\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.161619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.161569    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389006f4-6c3d-4519-8744-46882675e05b-istio-podinfo\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.161679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.161660    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xfzj\" (UniqueName: \"kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-kube-api-access-2xfzj\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.161827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.161810    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-envoy\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.263874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.263838    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-data\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.264045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.263912    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389006f4-6c3d-4519-8744-46882675e05b-istio-podinfo\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.264116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.264048    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xfzj\" (UniqueName: \"kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-kube-api-access-2xfzj\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.264298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.264278    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-envoy\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.264375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.264297    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-data\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.264499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.264463    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-credential-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.264499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.264491    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-istio-token\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.264593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.264524    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389006f4-6c3d-4519-8744-46882675e05b-istiod-ca-cert\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.264593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.264557    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.264702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.264621    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-certs\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.265024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.264999    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-certs\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.265419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.265377    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.265702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.265681    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-credential-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.266453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.266381    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389006f4-6c3d-4519-8744-46882675e05b-istiod-ca-cert\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.267672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.267627    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389006f4-6c3d-4519-8744-46882675e05b-istio-podinfo\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.267780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.267721    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-envoy\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.273673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.273609    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-istio-token\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.273989 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.273973    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xfzj\" (UniqueName: \"kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-kube-api-access-2xfzj\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.343995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.343964    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:36.621947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.621861    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" event={"ID":"6afa42af-0e54-4c13-80fa-e0d66c519b4a","Type":"ContainerStarted","Data":"2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af"}
Oct 30 21:33:36.621947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.621914    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" event={"ID":"6afa42af-0e54-4c13-80fa-e0d66c519b4a","Type":"ContainerStarted","Data":"c8dc99f788e0eea1c1af78df0ae9c3c302edb65a21bbaf0d5c536674a82210bf"}
Oct 30 21:33:36.623705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.623678    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" event={"ID":"5ec92974-c36a-4577-b40d-685b6d039c7d","Type":"ContainerStarted","Data":"0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e"}
Oct 30 21:33:36.623852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.623721    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" event={"ID":"5ec92974-c36a-4577-b40d-685b6d039c7d","Type":"ContainerStarted","Data":"c054804a11f1193418a36adc1b8e34f0ed62a864aae82a263cd76de63b78b3dd"}
Oct 30 21:33:36.639315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.639296    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"]
Oct 30 21:33:36.642080 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:33:36.642044    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389006f4_6c3d_4519_8744_46882675e05b.slice/crio-94b39ea922aff009c83a659d64ee5681164bffac0572fdef453caa1b2a22f17b WatchSource:0}: Error finding container 94b39ea922aff009c83a659d64ee5681164bffac0572fdef453caa1b2a22f17b: Status 404 returned error can't find the container with id 94b39ea922aff009c83a659d64ee5681164bffac0572fdef453caa1b2a22f17b
Oct 30 21:33:36.646235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.646175    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:33:36.646334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.646293    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:33:36.646385 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.646341    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:33:36.653315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:36.653266    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" podStartSLOduration=2.653248261 podStartE2EDuration="2.653248261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:36.645434362 +0000 UTC m=+1929.691993525" watchObservedRunningTime="2026-10-30 21:33:36.653248261 +0000 UTC m=+1929.699807406"
Oct 30 21:33:37.370663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:37.370588    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:37.372254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:37.372232    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 21:33:37.372346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:37.372282    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" podUID="5ec92974-c36a-4577-b40d-685b6d039c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 21:33:37.409440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:37.407179    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:37.416584 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:37.416547    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 21:33:37.416891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:37.416848    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" podUID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 21:33:37.651855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:37.651762    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" event={"ID":"389006f4-6c3d-4519-8744-46882675e05b","Type":"ContainerStarted","Data":"992bd4ec2e39a493c4b8d71b634628a72b5c94289ecf2771a9630fe3677dc07f"}
Oct 30 21:33:37.651855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:37.651824    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" event={"ID":"389006f4-6c3d-4519-8744-46882675e05b","Type":"ContainerStarted","Data":"94b39ea922aff009c83a659d64ee5681164bffac0572fdef453caa1b2a22f17b"}
Oct 30 21:33:37.679416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:37.679362    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" podStartSLOduration=3.67934924 podStartE2EDuration="3.67934924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:36.689731616 +0000 UTC m=+1929.736290793" watchObservedRunningTime="2026-10-30 21:33:37.67934924 +0000 UTC m=+1930.725908385"
Oct 30 21:33:37.680716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:37.680521    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" podStartSLOduration=2.680510564 podStartE2EDuration="2.680510564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:37.672607573 +0000 UTC m=+1930.719166733" watchObservedRunningTime="2026-10-30 21:33:37.680510564 +0000 UTC m=+1930.727069714"
Oct 30 21:33:38.366858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:38.366818    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:38.370687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:38.370591    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 21:33:38.370687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:38.370654    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" podUID="5ec92974-c36a-4577-b40d-685b6d039c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 21:33:38.372402 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:38.372347    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 21:33:38.372542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:38.372423    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" podUID="389006f4-6c3d-4519-8744-46882675e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 21:33:38.406418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:38.404808    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 21:33:38.406418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:38.404867    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" podUID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 21:33:39.829416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:39.793470    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 21:33:39.829416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:39.793529    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" podUID="5ec92974-c36a-4577-b40d-685b6d039c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 21:33:39.829416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:39.793955    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 21:33:39.829416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:39.793997    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" podUID="389006f4-6c3d-4519-8744-46882675e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 21:33:39.829416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:39.794069    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 21:33:39.829416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:39.794093    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" podUID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 21:33:40.071624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:40.067140    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:33:40.098478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:40.086366    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:33:40.147424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:40.146494    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:33:40.231745 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:33:40.231705    4918 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.12s"
Oct 30 21:33:40.370066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:40.369962    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 21:33:40.370429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:40.370369    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" podUID="389006f4-6c3d-4519-8744-46882675e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 21:33:40.380923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:40.380887    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 21:33:40.381213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:40.381187    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" podUID="5ec92974-c36a-4577-b40d-685b6d039c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 21:33:40.414350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:40.414316    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 21:33:40.415742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:40.415710    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" podUID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 21:33:40.572801 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:40.572741    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:33:41.346424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:41.345766    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 21:33:41.346424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:41.345836    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" podUID="389006f4-6c3d-4519-8744-46882675e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 21:33:41.390422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:41.382789    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 21:33:41.390422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:41.382857    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" podUID="5ec92974-c36a-4577-b40d-685b6d039c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 21:33:41.406025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:41.405973    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 21:33:41.406207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:41.406038    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" podUID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 21:33:42.377216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:42.377178    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body=
Oct 30 21:33:42.377811 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:42.377626    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" podUID="5ec92974-c36a-4577-b40d-685b6d039c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused"
Oct 30 21:33:42.378009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:42.377469    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 21:33:42.378175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:42.378154    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" podUID="389006f4-6c3d-4519-8744-46882675e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 21:33:42.411025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:42.410987    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body=
Oct 30 21:33:42.411299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:42.411273    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" podUID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused"
Oct 30 21:33:43.344674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:43.344624    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 21:33:43.344900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:43.344698    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" podUID="389006f4-6c3d-4519-8744-46882675e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 21:33:43.376826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:43.376795    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:43.377265 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:43.377251    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:43.378093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:43.378074    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:33:43.420959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:43.420930    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:43.421219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:43.421200    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:43.421978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:43.421958    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:33:44.344370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:44.344330    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body=
Oct 30 21:33:44.344610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:44.344419    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" podUID="389006f4-6c3d-4519-8744-46882675e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused"
Oct 30 21:33:45.457995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:45.457970    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:45.460647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:45.460554    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:45.461881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:45.461843    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:33:47.692416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:47.687128    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"]
Oct 30 21:33:47.720414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:47.717728    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-02ecd7-k26wp"]
Oct 30 21:33:49.081026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:49.080994    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3" path="/var/lib/kubelet/pods/aead39d9-7bdd-4f97-a9c7-239c0fdbd1a3/volumes"
Oct 30 21:33:55.197515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:55.197467    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"]
Oct 30 21:33:55.198136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:55.197740    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" podUID="5ec92974-c36a-4577-b40d-685b6d039c7d" containerName="istio-proxy" containerID="cri-o://0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e" gracePeriod=30
Oct 30 21:33:55.203131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:55.203105    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"]
Oct 30 21:33:55.203340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:55.203319    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" podUID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" containerName="istio-proxy" containerID="cri-o://2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af" gracePeriod=30
Oct 30 21:33:58.379310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:58.379277    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 21:33:58.379743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:58.379352    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" podUID="5ec92974-c36a-4577-b40d-685b6d039c7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:33:58.423317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:58.423288    4918 patch_prober.go:29] interesting pod/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:33:58.423501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:33:58.423339    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" podUID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:33:59.163273 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:33:59.163231    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:34:00.523896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.523868    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:34:00.528787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.528758    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:34:00.615617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.615585    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-envoy\") pod \"5ec92974-c36a-4577-b40d-685b6d039c7d\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") "
Oct 30 21:34:00.615810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.615646    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-kube-api-access-qbm5b\" (UniqueName: \"kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-kube-api-access-qbm5b\") pod \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") "
Oct 30 21:34:00.615810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.615717    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-certs\") pod \"5ec92974-c36a-4577-b40d-685b6d039c7d\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") "
Oct 30 21:34:00.615943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.615831    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-socket\") pod \"5ec92974-c36a-4577-b40d-685b6d039c7d\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") "
Oct 30 21:34:00.615943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.615874    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-certs\") pod \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") "
Oct 30 21:34:00.615943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.615915    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-data\") pod \"5ec92974-c36a-4577-b40d-685b6d039c7d\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") "
Oct 30 21:34:00.616116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.615951    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-podinfo\") pod \"5ec92974-c36a-4577-b40d-685b6d039c7d\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") "
Oct 30 21:34:00.616116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.615989    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-kube-api-access-gbflk\" (UniqueName: \"kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-kube-api-access-gbflk\") pod \"5ec92974-c36a-4577-b40d-685b6d039c7d\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") "
Oct 30 21:34:00.616116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.616013    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-certs" pod "5ec92974-c36a-4577-b40d-685b6d039c7d" (UID: "5ec92974-c36a-4577-b40d-685b6d039c7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:00.616116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.616037    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istiod-ca-cert\") pod \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") "
Oct 30 21:34:00.616335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.616117    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-podinfo\") pod \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") "
Oct 30 21:34:00.616335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.616174    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-credential-socket\") pod \"5ec92974-c36a-4577-b40d-685b6d039c7d\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") "
Oct 30 21:34:00.616335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.616206    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-socket\") pod \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") "
Oct 30 21:34:00.616335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.616222    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-certs" pod "6afa42af-0e54-4c13-80fa-e0d66c519b4a" (UID: "6afa42af-0e54-4c13-80fa-e0d66c519b4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:00.616335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.616283    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec92974-c36a-4577-b40d-685b6d039c7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec92974-c36a-4577-b40d-685b6d039c7d-istiod-ca-cert\") pod \"5ec92974-c36a-4577-b40d-685b6d039c7d\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") "
Oct 30 21:34:00.616335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.616320    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-token\" (UniqueName: \"kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-token\") pod \"5ec92974-c36a-4577-b40d-685b6d039c7d\" (UID: \"5ec92974-c36a-4577-b40d-685b6d039c7d\") "
Oct 30 21:34:00.616695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.616460    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-token\" (UniqueName: \"kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-token\") pod \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") "
Oct 30 21:34:00.616789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.616711    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-credential-socket\") pod \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") "
Oct 30 21:34:00.616855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.616826    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-envoy\") pod \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") "
Oct 30 21:34:00.616918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.616876    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-data\") pod \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\" (UID: \"6afa42af-0e54-4c13-80fa-e0d66c519b4a\") "
Oct 30 21:34:00.620671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.617166    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-socket" pod "5ec92974-c36a-4577-b40d-685b6d039c7d" (UID: "5ec92974-c36a-4577-b40d-685b6d039c7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:00.620805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.617347    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-socket" pod "6afa42af-0e54-4c13-80fa-e0d66c519b4a" (UID: "6afa42af-0e54-4c13-80fa-e0d66c519b4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:00.620805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.617480    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-data" pod "5ec92974-c36a-4577-b40d-685b6d039c7d" (UID: "5ec92974-c36a-4577-b40d-685b6d039c7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:00.620805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.617593    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istiod-ca-cert" pod "6afa42af-0e54-4c13-80fa-e0d66c519b4a" (UID: "6afa42af-0e54-4c13-80fa-e0d66c519b4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:00.620805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.617658    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-credential-socket" pod "5ec92974-c36a-4577-b40d-685b6d039c7d" (UID: "5ec92974-c36a-4577-b40d-685b6d039c7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:00.620805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.617952    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec92974-c36a-4577-b40d-685b6d039c7d-istiod-ca-cert" pod "5ec92974-c36a-4577-b40d-685b6d039c7d" (UID: "5ec92974-c36a-4577-b40d-685b6d039c7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:00.620805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.618155    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-credential-socket" pod "6afa42af-0e54-4c13-80fa-e0d66c519b4a" (UID: "6afa42af-0e54-4c13-80fa-e0d66c519b4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:00.620805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.619373    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-envoy" pod "5ec92974-c36a-4577-b40d-685b6d039c7d" (UID: "5ec92974-c36a-4577-b40d-685b6d039c7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:00.620805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.619958    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-podinfo" pod "5ec92974-c36a-4577-b40d-685b6d039c7d" (UID: "5ec92974-c36a-4577-b40d-685b6d039c7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:34:00.620805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.620192    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-data" pod "6afa42af-0e54-4c13-80fa-e0d66c519b4a" (UID: "6afa42af-0e54-4c13-80fa-e0d66c519b4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:00.620805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.620382    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-kube-api-access-qbm5b" pod "6afa42af-0e54-4c13-80fa-e0d66c519b4a" (UID: "6afa42af-0e54-4c13-80fa-e0d66c519b4a"). InnerVolumeSpecName "kube-api-access-qbm5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:00.621667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621642    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-podinfo" pod "6afa42af-0e54-4c13-80fa-e0d66c519b4a" (UID: "6afa42af-0e54-4c13-80fa-e0d66c519b4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:34:00.621764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621653    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.621764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621692    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.621764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621708    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbm5b\" (UniqueName: \"kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-kube-api-access-qbm5b\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.621764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621721    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.621764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621734    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.621764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621746    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.621764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621757    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.622176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621770    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.622176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621783    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.622176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621807    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec92974-c36a-4577-b40d-685b6d039c7d-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.622176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621809    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-token" pod "6afa42af-0e54-4c13-80fa-e0d66c519b4a" (UID: "6afa42af-0e54-4c13-80fa-e0d66c519b4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:00.622176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621820    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.622176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621837    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec92974-c36a-4577-b40d-685b6d039c7d-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.622176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621850    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.622176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.621981    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-kube-api-access-gbflk" pod "5ec92974-c36a-4577-b40d-685b6d039c7d" (UID: "5ec92974-c36a-4577-b40d-685b6d039c7d"). InnerVolumeSpecName "kube-api-access-gbflk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:00.623319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.623298    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-envoy" pod "6afa42af-0e54-4c13-80fa-e0d66c519b4a" (UID: "6afa42af-0e54-4c13-80fa-e0d66c519b4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:00.623710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.623693    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-token" pod "5ec92974-c36a-4577-b40d-685b6d039c7d" (UID: "5ec92974-c36a-4577-b40d-685b6d039c7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:00.759690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.752406    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbflk\" (UniqueName: \"kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-kube-api-access-gbflk\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.759690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.752450    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.759690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.752466    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec92974-c36a-4577-b40d-685b6d039c7d-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.759690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.752480    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:00.759690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:00.752496    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afa42af-0e54-4c13-80fa-e0d66c519b4a-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:34:01.073730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.073635    4918 generic.go:356] "Generic (PLEG): container finished" podID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" containerID="2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af" exitCode=0
Oct 30 21:34:01.073972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.073805    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"
Oct 30 21:34:01.084633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.084597    4918 generic.go:356] "Generic (PLEG): container finished" podID="5ec92974-c36a-4577-b40d-685b6d039c7d" containerID="0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e" exitCode=0
Oct 30 21:34:01.084818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.084718    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"
Oct 30 21:34:01.098947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.098911    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" event={"ID":"6afa42af-0e54-4c13-80fa-e0d66c519b4a","Type":"ContainerDied","Data":"2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af"}
Oct 30 21:34:01.099063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.098950    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h" event={"ID":"6afa42af-0e54-4c13-80fa-e0d66c519b4a","Type":"ContainerDied","Data":"c8dc99f788e0eea1c1af78df0ae9c3c302edb65a21bbaf0d5c536674a82210bf"}
Oct 30 21:34:01.099063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.098966    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" event={"ID":"5ec92974-c36a-4577-b40d-685b6d039c7d","Type":"ContainerDied","Data":"0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e"}
Oct 30 21:34:01.099063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.098980    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb" event={"ID":"5ec92974-c36a-4577-b40d-685b6d039c7d","Type":"ContainerDied","Data":"c054804a11f1193418a36adc1b8e34f0ed62a864aae82a263cd76de63b78b3dd"}
Oct 30 21:34:01.099063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.099002    4918 scope.go:122] "RemoveContainer" containerID="2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af"
Oct 30 21:34:01.116896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.115428    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"]
Oct 30 21:34:01.116896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.115483    4918 scope.go:122] "RemoveContainer" containerID="2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af"
Oct 30 21:34:01.116896 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:34:01.116593    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af\": container with ID starting with 2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af not found: ID does not exist" containerID="2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af"
Oct 30 21:34:01.116896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.116626    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af"} err="failed to get container status \"2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af\": rpc error: code = NotFound desc = could not find container \"2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af\": container with ID starting with 2f0119fde63f1b0b55b48f4c156f7fa0dcdf8be8d27072eed3501d41946cd7af not found: ID does not exist"
Oct 30 21:34:01.116896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.116647    4918 scope.go:122] "RemoveContainer" containerID="0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e"
Oct 30 21:34:01.123630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.123608    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-vtm7h"]
Oct 30 21:34:01.133258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.133237    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"]
Oct 30 21:34:01.134123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.134103    4918 scope.go:122] "RemoveContainer" containerID="0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e"
Oct 30 21:34:01.134487 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:34:01.134461    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e\": container with ID starting with 0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e not found: ID does not exist" containerID="0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e"
Oct 30 21:34:01.134571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.134500    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e"} err="failed to get container status \"0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e\": rpc error: code = NotFound desc = could not find container \"0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e\": container with ID starting with 0a288f233447e3cf6e9345a0742938aa5908ecabc6357ac371f6593a7e67c21e not found: ID does not exist"
Oct 30 21:34:01.138954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:01.138932    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-02ecd7-openshift-default-6644d8b8c7-8x9gb"]
Oct 30 21:34:03.081825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:03.081790    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec92974-c36a-4577-b40d-685b6d039c7d" path="/var/lib/kubelet/pods/5ec92974-c36a-4577-b40d-685b6d039c7d/volumes"
Oct 30 21:34:03.082305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:03.082282    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" path="/var/lib/kubelet/pods/6afa42af-0e54-4c13-80fa-e0d66c519b4a/volumes"
Oct 30 21:34:21.066705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.066675    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"]
Oct 30 21:34:21.067469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.067453    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5ec92974-c36a-4577-b40d-685b6d039c7d" containerName="istio-proxy"
Oct 30 21:34:21.067525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.067471    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec92974-c36a-4577-b40d-685b6d039c7d" containerName="istio-proxy"
Oct 30 21:34:21.067525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.067488    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" containerName="istio-proxy"
Oct 30 21:34:21.067525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.067494    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" containerName="istio-proxy"
Oct 30 21:34:21.067659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.067649    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec92974-c36a-4577-b40d-685b6d039c7d" containerName="istio-proxy"
Oct 30 21:34:21.067695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.067661    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6afa42af-0e54-4c13-80fa-e0d66c519b4a" containerName="istio-proxy"
Oct 30 21:34:21.071447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.071408    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.085598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.085570    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"]
Oct 30 21:34:21.139568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.139530    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.139798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.139771    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-podinfo\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.139993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.139968    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-credential-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.140100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.140076    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-envoy\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.140272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.140233    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-certs\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.140441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.140405    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istiod-ca-cert\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.140541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.140439    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qldxl\" (UniqueName: \"kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-kube-api-access-qldxl\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.140608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.140590    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-data\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.140670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.140653    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-token\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.242764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.242721    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.242927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.242822    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-podinfo\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.243024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.242929    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-credential-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.243024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.242986    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-envoy\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.243142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.243089    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-certs\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.243200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.243173    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istiod-ca-cert\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.243200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.243158    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.243314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.243244    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qldxl\" (UniqueName: \"kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-kube-api-access-qldxl\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.243380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.243314    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-data\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.243380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.243323    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-credential-socket\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.243568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.243495    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-certs\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.243731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.243703    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-token\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.244028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.243713    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-data\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.244028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.243913    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istiod-ca-cert\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.245655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.245627    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-envoy\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.245744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.245670    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-podinfo\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.254980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.254956    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qldxl\" (UniqueName: \"kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-kube-api-access-qldxl\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.255370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.255348    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-token\") pod \"e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") " pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.393264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.393165    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:21.761678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.761649    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"]
Oct 30 21:34:21.762020 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:34:21.761998    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c890ea5_9ab7_4c1f_86bc_a13c55e9e870.slice/crio-14ad47c6f938611f3fe5051e370d83576159093aeea93187e97b866b94d37541 WatchSource:0}: Error finding container 14ad47c6f938611f3fe5051e370d83576159093aeea93187e97b866b94d37541: Status 404 returned error can't find the container with id 14ad47c6f938611f3fe5051e370d83576159093aeea93187e97b866b94d37541
Oct 30 21:34:21.764986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.764950    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:34:21.765078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.765015    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:34:21.765078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:21.765048    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:34:22.349990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:22.349951    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" event={"ID":"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870","Type":"ContainerStarted","Data":"4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37"}
Oct 30 21:34:22.350420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:22.350044    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" event={"ID":"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870","Type":"ContainerStarted","Data":"14ad47c6f938611f3fe5051e370d83576159093aeea93187e97b866b94d37541"}
Oct 30 21:34:22.385415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:22.377165    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" podStartSLOduration=1.377147625 podStartE2EDuration="1.377147625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:22.37271228 +0000 UTC m=+1975.419271426" watchObservedRunningTime="2026-10-30 21:34:22.377147625 +0000 UTC m=+1975.423706771"
Oct 30 21:34:22.413415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:22.412585    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:22.415049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:22.415012    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 21:34:22.415204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:22.415080    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" podUID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 21:34:23.402415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:23.396892    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 21:34:23.402415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:23.396952    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" podUID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 21:34:24.401442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:24.395954    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 21:34:24.401442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:24.396031    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" podUID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 21:34:25.397790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:25.397748    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 21:34:25.400730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:25.400694    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" podUID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 21:34:26.395600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:26.395564    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 21:34:26.403534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:26.403491    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" podUID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 21:34:27.410419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:27.409499    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 21:34:27.410419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:27.409604    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" podUID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 21:34:28.394297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:28.394264    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body=
Oct 30 21:34:28.394527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:28.394315    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" podUID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused"
Oct 30 21:34:29.161243 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:34:29.161206    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:34:29.398720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:29.398692    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:29.399117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:29.399089    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:29.399726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:29.399708    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:34:31.155683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:31.155651    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"]
Oct 30 21:34:31.160987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:31.160957    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"
Oct 30 21:34:31.179175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:31.179145    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"]
Oct 30 21:34:31.296453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:31.267086    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c4k4\" (UniqueName: \"kubernetes.io/projected/ddfed72d-ef13-4d74-9cae-cd8758f202a0-kube-api-access-5c4k4\") pod \"payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk\" (UID: \"ddfed72d-ef13-4d74-9cae-cd8758f202a0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"
Oct 30 21:34:31.323777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:31.323740    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfed72d-ef13-4d74-9cae-cd8758f202a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk\" (UID: \"ddfed72d-ef13-4d74-9cae-cd8758f202a0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"
Oct 30 21:34:31.336409 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:31.312621    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-f9e5a7-dockercfg-8d6wc\""
Oct 30 21:34:31.336673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:31.312349    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-f9e5a7\""
Oct 30 21:34:31.429596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:31.429503    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfed72d-ef13-4d74-9cae-cd8758f202a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk\" (UID: \"ddfed72d-ef13-4d74-9cae-cd8758f202a0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"
Oct 30 21:34:31.448994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:31.448951    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c4k4\" (UniqueName: \"kubernetes.io/projected/ddfed72d-ef13-4d74-9cae-cd8758f202a0-kube-api-access-5c4k4\") pod \"payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk\" (UID: \"ddfed72d-ef13-4d74-9cae-cd8758f202a0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"
Oct 30 21:34:31.468418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:31.451424    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfed72d-ef13-4d74-9cae-cd8758f202a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk\" (UID: \"ddfed72d-ef13-4d74-9cae-cd8758f202a0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"
Oct 30 21:34:31.509418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:31.507300    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c4k4\" (UniqueName: \"kubernetes.io/projected/ddfed72d-ef13-4d74-9cae-cd8758f202a0-kube-api-access-5c4k4\") pod \"payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk\" (UID: \"ddfed72d-ef13-4d74-9cae-cd8758f202a0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"
Oct 30 21:34:31.819521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:31.819487    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"
Oct 30 21:34:32.136281 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:34:32.136244    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfed72d_ef13_4d74_9cae_cd8758f202a0.slice/crio-4e6b9538ebfa9769d32e63bde5705d7f45b3429fdebdc43754d28f2c36809852 WatchSource:0}: Error finding container 4e6b9538ebfa9769d32e63bde5705d7f45b3429fdebdc43754d28f2c36809852: Status 404 returned error can't find the container with id 4e6b9538ebfa9769d32e63bde5705d7f45b3429fdebdc43754d28f2c36809852
Oct 30 21:34:32.136824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:32.136801    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"]
Oct 30 21:34:32.707457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:32.706920    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk" event={"ID":"ddfed72d-ef13-4d74-9cae-cd8758f202a0","Type":"ContainerStarted","Data":"5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72"}
Oct 30 21:34:32.707457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:32.706987    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk" event={"ID":"ddfed72d-ef13-4d74-9cae-cd8758f202a0","Type":"ContainerStarted","Data":"4e6b9538ebfa9769d32e63bde5705d7f45b3429fdebdc43754d28f2c36809852"}
Oct 30 21:34:32.711915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:32.708359    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"
Oct 30 21:34:32.733848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:32.733791    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk" podStartSLOduration=1.733777184 podStartE2EDuration="1.733777184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:32.728194633 +0000 UTC m=+1985.774753802" watchObservedRunningTime="2026-10-30 21:34:32.733777184 +0000 UTC m=+1985.780336328"
Oct 30 21:34:43.601926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.601891    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"]
Oct 30 21:34:43.607956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.607923    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.616460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.616433    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-36a099-openshift-default-dockercfg-cv5vg\""
Oct 30 21:34:43.623825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.623801    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"]
Oct 30 21:34:43.774412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.768888    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-credential-socket\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.774412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.768956    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2jbj\" (UniqueName: \"kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-kube-api-access-d2jbj\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.774412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.768990    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-socket\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.774412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.769084    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-certs\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.774412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.769119    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5b070d-b6a5-4eb0-be88-300467b249f9-istiod-ca-cert\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.774412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.769219    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-token\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.774412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.769285    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-data\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.774412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.769371    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-envoy\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.774412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.769447    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-podinfo\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.903467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.898475    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5b070d-b6a5-4eb0-be88-300467b249f9-istiod-ca-cert\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.903467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.898595    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-token\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.903467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.898657    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-data\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.903467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.898739    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-envoy\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.903467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.898853    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-podinfo\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.903467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.899119    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-credential-socket\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.903467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.899173    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jbj\" (UniqueName: \"kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-kube-api-access-d2jbj\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.903467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.899206    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-socket\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.903467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.899291    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-certs\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.903467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.899809    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-certs\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.918414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.916341    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-data\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.918414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.916728    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-credential-socket\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.918414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.916956    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-socket\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.918414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.917249    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5b070d-b6a5-4eb0-be88-300467b249f9-istiod-ca-cert\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.940416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.931800    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-envoy\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.968440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.963146    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-token\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:43.984472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:43.983584    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-podinfo\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:44.004416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:44.000780    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jbj\" (UniqueName: \"kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-kube-api-access-d2jbj\") pod \"e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") " pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:44.230199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:44.230169    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:44.714062 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:34:44.713868    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe5b070d_b6a5_4eb0_be88_300467b249f9.slice/crio-70962f7c8aa163a8c96f4c60093135816c093303ff597af13c0376aa15b20d63 WatchSource:0}: Error finding container 70962f7c8aa163a8c96f4c60093135816c093303ff597af13c0376aa15b20d63: Status 404 returned error can't find the container with id 70962f7c8aa163a8c96f4c60093135816c093303ff597af13c0376aa15b20d63
Oct 30 21:34:44.736416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:44.716856    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"]
Oct 30 21:34:44.736780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:44.720945    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:34:44.737095 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:44.736956    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:34:44.741412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:44.737329    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:34:44.783730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:44.782748    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"
Oct 30 21:34:44.907149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:44.907112    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" event={"ID":"be5b070d-b6a5-4eb0-be88-300467b249f9","Type":"ContainerStarted","Data":"70962f7c8aa163a8c96f4c60093135816c093303ff597af13c0376aa15b20d63"}
Oct 30 21:34:45.192171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.192135    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"]
Oct 30 21:34:45.200376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.200322    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.208306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.208273    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"]
Oct 30 21:34:45.208672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.208651    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"agc-cred-e7adf1e2358d\""
Oct 30 21:34:45.208794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.208743    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"agc-cred-eceb81ede848\""
Oct 30 21:34:45.208986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.208940    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"agc-cred-c07a671227da\""
Oct 30 21:34:45.209070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.209038    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"routing-overlay\""
Oct 30 21:34:45.209122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.209072    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"payload-processing-external-model-e2e--650b641a-dockercfg-s6bl5\""
Oct 30 21:34:45.209323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.209305    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"openshift-service-ca.crt\""
Oct 30 21:34:45.209323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.209312    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f1ed97\""
Oct 30 21:34:45.210472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.210448    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"kube-root-ca.crt\""
Oct 30 21:34:45.302153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.302120    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.302328 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.302267    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.302328 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.302309    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.302491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.302474    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcm4p\" (UniqueName: \"kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-kube-api-access-tcm4p\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.407440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.404606    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcm4p\" (UniqueName: \"kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-kube-api-access-tcm4p\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.407440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.404891    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.407440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.405020    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.407440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.405063    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.410069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.408192    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.410069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.409828    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.431833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.428370    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcm4p\" (UniqueName: \"kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-kube-api-access-tcm4p\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.431833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.431595    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.522690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.522649    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.806483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.806447    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"]
Oct 30 21:34:45.806883 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:34:45.806851    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35743d11_427d_46dd_a9d5_fa08373a702b.slice/crio-8170339598c7e42b9aa436bdbd26205a3100ad179f0b2be8ed2b2283ae3e1763 WatchSource:0}: Error finding container 8170339598c7e42b9aa436bdbd26205a3100ad179f0b2be8ed2b2283ae3e1763: Status 404 returned error can't find the container with id 8170339598c7e42b9aa436bdbd26205a3100ad179f0b2be8ed2b2283ae3e1763
Oct 30 21:34:45.918385 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.918348    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" event={"ID":"be5b070d-b6a5-4eb0-be88-300467b249f9","Type":"ContainerStarted","Data":"51dc0ec76e9d0e56db1226c4800fe338cc898383a972123c56422222bab05250"}
Oct 30 21:34:45.920372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.920346    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw" event={"ID":"35743d11-427d-46dd-a9d5-fa08373a702b","Type":"ContainerStarted","Data":"b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00"}
Oct 30 21:34:45.920550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.920424    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw" event={"ID":"35743d11-427d-46dd-a9d5-fa08373a702b","Type":"ContainerStarted","Data":"8170339598c7e42b9aa436bdbd26205a3100ad179f0b2be8ed2b2283ae3e1763"}
Oct 30 21:34:45.920550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.920454    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:45.947790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.947743    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" podStartSLOduration=2.947725894 podStartE2EDuration="2.947725894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:45.94164903 +0000 UTC m=+1998.988208208" watchObservedRunningTime="2026-10-30 21:34:45.947725894 +0000 UTC m=+1998.994285039"
Oct 30 21:34:45.965867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:45.965812    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw" podStartSLOduration=0.96578767 podStartE2EDuration="965.78767ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:45.959592464 +0000 UTC m=+1999.006151628" watchObservedRunningTime="2026-10-30 21:34:45.96578767 +0000 UTC m=+1999.012346816"
Oct 30 21:34:46.236417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:46.232425    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:46.256287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:46.255989    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:46.930361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:46.930328    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:46.931289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:46.931265    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:34:49.881419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:49.881364    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"]
Oct 30 21:34:49.886577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:49.886555    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"
Oct 30 21:34:49.897577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:49.897530    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-36a099\""
Oct 30 21:34:49.899128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:49.899102    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"]
Oct 30 21:34:50.140519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:50.134838    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v\" (UID: \"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"
Oct 30 21:34:50.140519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:50.135253    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lckvt\" (UniqueName: \"kubernetes.io/projected/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-kube-api-access-lckvt\") pod \"payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v\" (UID: \"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"
Oct 30 21:34:50.252420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:50.241838    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v\" (UID: \"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"
Oct 30 21:34:50.252420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:50.242152    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lckvt\" (UniqueName: \"kubernetes.io/projected/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-kube-api-access-lckvt\") pod \"payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v\" (UID: \"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"
Oct 30 21:34:50.252420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:50.243381    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v\" (UID: \"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"
Oct 30 21:34:50.328412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:50.321237    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lckvt\" (UniqueName: \"kubernetes.io/projected/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-kube-api-access-lckvt\") pod \"payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v\" (UID: \"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"
Oct 30 21:34:50.506637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:50.506608    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"
Oct 30 21:34:50.871470 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:34:50.866998    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ef6ad5_f0ac_4096_8682_bfcbb76437a3.slice/crio-e5718cd30b764bab7cc1cf3b804119ad9374e1b5fef709b7370d1a96f2571767 WatchSource:0}: Error finding container e5718cd30b764bab7cc1cf3b804119ad9374e1b5fef709b7370d1a96f2571767: Status 404 returned error can't find the container with id e5718cd30b764bab7cc1cf3b804119ad9374e1b5fef709b7370d1a96f2571767
Oct 30 21:34:50.873529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:50.873497    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"]
Oct 30 21:34:51.046746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:51.046713    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v" event={"ID":"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3","Type":"ContainerStarted","Data":"e5718cd30b764bab7cc1cf3b804119ad9374e1b5fef709b7370d1a96f2571767"}
Oct 30 21:34:52.056991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:52.056940    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v" event={"ID":"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3","Type":"ContainerStarted","Data":"d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969"}
Oct 30 21:34:52.057480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:52.057037    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"
Oct 30 21:34:52.076770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:52.076726    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v" podStartSLOduration=3.076712816 podStartE2EDuration="3.076712816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:52.072491242 +0000 UTC m=+2005.119050414" watchObservedRunningTime="2026-10-30 21:34:52.076712816 +0000 UTC m=+2005.123272002"
Oct 30 21:34:54.306065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.306031    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"]
Oct 30 21:34:54.311525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.311504    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.322735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.322713    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a378a2-openshift-default-dockercfg-4ch9v\""
Oct 30 21:34:54.324974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.324952    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"]
Oct 30 21:34:54.424336 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.424297    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istiod-ca-cert\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.424669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.424385    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-envoy\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.424669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.424430    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.424669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.424460    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-data\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.424669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.424565    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-token\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.424669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.424602    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfq9f\" (UniqueName: \"kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-kube-api-access-pfq9f\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.424669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.424641    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-certs\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.425097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.424731    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-credential-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.425097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.424800    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-podinfo\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.526377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.526337    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-token\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.526563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.526418    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfq9f\" (UniqueName: \"kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-kube-api-access-pfq9f\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.526563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.526486    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-certs\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.526691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.526590    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-credential-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.526691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.526663    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-podinfo\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.526813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.526788    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istiod-ca-cert\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.526893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.526877    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-envoy\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.526950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.526910    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.526950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.526941    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-data\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.527063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.526994    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-certs\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.527063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.527042    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-credential-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.527330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.527300    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.527330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.527326    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-data\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.527763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.527509    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istiod-ca-cert\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.529483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.529461    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-podinfo\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.529584 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.529501    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-envoy\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.538669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.538637    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-token\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.548018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.547995    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfq9f\" (UniqueName: \"kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-kube-api-access-pfq9f\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.629293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.629217    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:54.925260 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:34:54.925232    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c41b0e_4cdb_47e3_aa79_c5b22a123c62.slice/crio-34a5c27276e1e0186635af3931a44c9709f8edee7682d5979c86a940ad01a5ee WatchSource:0}: Error finding container 34a5c27276e1e0186635af3931a44c9709f8edee7682d5979c86a940ad01a5ee: Status 404 returned error can't find the container with id 34a5c27276e1e0186635af3931a44c9709f8edee7682d5979c86a940ad01a5ee
Oct 30 21:34:54.925966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.925945    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"]
Oct 30 21:34:54.928423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.928356    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:34:54.928512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.928462    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:34:54.928512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:54.928494    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:34:55.085762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:55.085727    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5" event={"ID":"24c41b0e-4cdb-47e3-aa79-c5b22a123c62","Type":"ContainerStarted","Data":"c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a"}
Oct 30 21:34:55.085943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:55.085786    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5" event={"ID":"24c41b0e-4cdb-47e3-aa79-c5b22a123c62","Type":"ContainerStarted","Data":"34a5c27276e1e0186635af3931a44c9709f8edee7682d5979c86a940ad01a5ee"}
Oct 30 21:34:55.112207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:55.112159    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5" podStartSLOduration=1.112144227 podStartE2EDuration="1.112144227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:55.109079386 +0000 UTC m=+2008.155638532" watchObservedRunningTime="2026-10-30 21:34:55.112144227 +0000 UTC m=+2008.158703371"
Oct 30 21:34:55.629692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:55.629655    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:34:56.631893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:56.631854    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body=
Oct 30 21:34:56.632293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:56.631915    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5" podUID="24c41b0e-4cdb-47e3-aa79-c5b22a123c62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused"
Oct 30 21:34:56.932978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:56.932886    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:34:57.629721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:57.629680    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body=
Oct 30 21:34:57.630087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:57.629757    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5" podUID="24c41b0e-4cdb-47e3-aa79-c5b22a123c62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused"
Oct 30 21:34:58.630047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:58.630015    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body=
Oct 30 21:34:58.630465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:58.630077    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5" podUID="24c41b0e-4cdb-47e3-aa79-c5b22a123c62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused"
Oct 30 21:34:59.161833 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:34:59.161788    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:34:59.639593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:59.639559    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body=
Oct 30 21:34:59.640444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:34:59.640382    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5" podUID="24c41b0e-4cdb-47e3-aa79-c5b22a123c62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused"
Oct 30 21:35:00.629772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:00.629731    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body=
Oct 30 21:35:00.629998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:00.629802    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5" podUID="24c41b0e-4cdb-47e3-aa79-c5b22a123c62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused"
Oct 30 21:35:01.635809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:01.635773    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:35:01.636315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:01.636169    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:35:01.636825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:01.636803    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:35:03.068790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:03.068751    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"
Oct 30 21:35:16.239275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.239196    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"]
Oct 30 21:35:16.243903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.243878    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:16.250531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.250200    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:35:16.250531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.250336    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:35:16.250531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.250461    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:35:16.250531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.250500    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:35:16.256282 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.256261    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"]
Oct 30 21:35:16.407308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.407271    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f088835f-j599x\" (UID: \"410f3d85-4a38-40fc-b478-85e06686f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:16.407534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.407318    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cjjz\" (UniqueName: \"kubernetes.io/projected/410f3d85-4a38-40fc-b478-85e06686f9b6-kube-api-access-2cjjz\") pod \"maas-api-revoke-sub-f088835f-j599x\" (UID: \"410f3d85-4a38-40fc-b478-85e06686f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:16.407597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.407539    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f088835f-j599x\" (UID: \"410f3d85-4a38-40fc-b478-85e06686f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:16.509975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.509852    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f088835f-j599x\" (UID: \"410f3d85-4a38-40fc-b478-85e06686f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:16.509975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.509920    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cjjz\" (UniqueName: \"kubernetes.io/projected/410f3d85-4a38-40fc-b478-85e06686f9b6-kube-api-access-2cjjz\") pod \"maas-api-revoke-sub-f088835f-j599x\" (UID: \"410f3d85-4a38-40fc-b478-85e06686f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:16.510263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.510071    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f088835f-j599x\" (UID: \"410f3d85-4a38-40fc-b478-85e06686f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:16.511252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.511216    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f088835f-j599x\" (UID: \"410f3d85-4a38-40fc-b478-85e06686f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:16.559788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.559747    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f088835f-j599x\" (UID: \"410f3d85-4a38-40fc-b478-85e06686f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:16.574471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.571271    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cjjz\" (UniqueName: \"kubernetes.io/projected/410f3d85-4a38-40fc-b478-85e06686f9b6-kube-api-access-2cjjz\") pod \"maas-api-revoke-sub-f088835f-j599x\" (UID: \"410f3d85-4a38-40fc-b478-85e06686f9b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:16.867459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:16.865562    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:17.418448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.418410    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"]
Oct 30 21:35:17.418893 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:35:17.418864    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410f3d85_4a38_40fc_b478_85e06686f9b6.slice/crio-c1fdf7718842422d29c3b2765b17920545053edba1dc2104aa677d55a5f64782 WatchSource:0}: Error finding container c1fdf7718842422d29c3b2765b17920545053edba1dc2104aa677d55a5f64782: Status 404 returned error can't find the container with id c1fdf7718842422d29c3b2765b17920545053edba1dc2104aa677d55a5f64782
Oct 30 21:35:17.702306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.701468    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x" event={"ID":"410f3d85-4a38-40fc-b478-85e06686f9b6","Type":"ContainerStarted","Data":"c1fdf7718842422d29c3b2765b17920545053edba1dc2104aa677d55a5f64782"}
Oct 30 21:35:17.788572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.788523    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"]
Oct 30 21:35:17.793714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.793693    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:17.801923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.801894    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"agc-cred-42957d623967\""
Oct 30 21:35:17.803266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.803246    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"agc-cred-d390aee96ac0\""
Oct 30 21:35:17.803560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.803538    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"payload-processing-external-model-e2e--8e1d66cd-dockercfg-h75qf\""
Oct 30 21:35:17.804859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.804841    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"openshift-service-ca.crt\""
Oct 30 21:35:17.804963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.804868    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"routing-overlay\""
Oct 30 21:35:17.804963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.804958    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"kube-root-ca.crt\""
Oct 30 21:35:17.805045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.804996    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea\""
Oct 30 21:35:17.831423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.831384    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"]
Oct 30 21:35:17.865154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.865115    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:17.865311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.865221    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:17.865477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.865426    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:17.865651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.865631    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54k89\" (UniqueName: \"kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-kube-api-access-54k89\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:17.967268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.967199    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54k89\" (UniqueName: \"kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-kube-api-access-54k89\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:17.967439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.967290    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:17.967439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.967340    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:17.967604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.967587    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:17.968126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.968100    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:17.968255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.968233    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:17.970324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.970296    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:17.978750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:17.978727    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54k89\" (UniqueName: \"kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-kube-api-access-54k89\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:18.114419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:18.111489    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:18.512982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:18.512957    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"]
Oct 30 21:35:18.550349 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:35:18.550320    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aed2990_6360_46eb_8322_07c97e074d2a.slice/crio-20bc909b5349ff3c956c95b28f35a93423805bf2c1be895f0c7adef5daffadf1 WatchSource:0}: Error finding container 20bc909b5349ff3c956c95b28f35a93423805bf2c1be895f0c7adef5daffadf1: Status 404 returned error can't find the container with id 20bc909b5349ff3c956c95b28f35a93423805bf2c1be895f0c7adef5daffadf1
Oct 30 21:35:18.712434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:18.712374    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d" event={"ID":"0aed2990-6360-46eb-8322-07c97e074d2a","Type":"ContainerStarted","Data":"65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257"}
Oct 30 21:35:18.712590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:18.712461    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:18.712590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:18.712485    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d" event={"ID":"0aed2990-6360-46eb-8322-07c97e074d2a","Type":"ContainerStarted","Data":"20bc909b5349ff3c956c95b28f35a93423805bf2c1be895f0c7adef5daffadf1"}
Oct 30 21:35:18.714341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:18.714316    4918 generic.go:356] "Generic (PLEG): container finished" podID="410f3d85-4a38-40fc-b478-85e06686f9b6" containerID="d11ded123281b7af2e7764ac92c4831ce21b036ecf5793d3c85e4eac9c86e14f" exitCode=0
Oct 30 21:35:18.714514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:18.714434    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x" event={"ID":"410f3d85-4a38-40fc-b478-85e06686f9b6","Type":"ContainerDied","Data":"d11ded123281b7af2e7764ac92c4831ce21b036ecf5793d3c85e4eac9c86e14f"}
Oct 30 21:35:18.745041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:18.744991    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d" podStartSLOduration=1.744977074 podStartE2EDuration="1.744977074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:18.740584071 +0000 UTC m=+2031.787143257" watchObservedRunningTime="2026-10-30 21:35:18.744977074 +0000 UTC m=+2031.791536197"
Oct 30 21:35:19.914758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:19.914731    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:19.942262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:19.942231    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-xks-ca\") pod \"410f3d85-4a38-40fc-b478-85e06686f9b6\" (UID: \"410f3d85-4a38-40fc-b478-85e06686f9b6\") "
Oct 30 21:35:19.942483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:19.942446    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-openshift-ca\") pod \"410f3d85-4a38-40fc-b478-85e06686f9b6\" (UID: \"410f3d85-4a38-40fc-b478-85e06686f9b6\") "
Oct 30 21:35:19.942594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:19.942577    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/410f3d85-4a38-40fc-b478-85e06686f9b6-kube-api-access-2cjjz\" (UniqueName: \"kubernetes.io/projected/410f3d85-4a38-40fc-b478-85e06686f9b6-kube-api-access-2cjjz\") pod \"410f3d85-4a38-40fc-b478-85e06686f9b6\" (UID: \"410f3d85-4a38-40fc-b478-85e06686f9b6\") "
Oct 30 21:35:19.942927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:19.942905    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-openshift-ca" pod "410f3d85-4a38-40fc-b478-85e06686f9b6" (UID: "410f3d85-4a38-40fc-b478-85e06686f9b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:19.943364 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:19.943325    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:19.944750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:19.944727    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-xks-ca" pod "410f3d85-4a38-40fc-b478-85e06686f9b6" (UID: "410f3d85-4a38-40fc-b478-85e06686f9b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:35:19.945318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:19.945283    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410f3d85-4a38-40fc-b478-85e06686f9b6-kube-api-access-2cjjz" pod "410f3d85-4a38-40fc-b478-85e06686f9b6" (UID: "410f3d85-4a38-40fc-b478-85e06686f9b6"). InnerVolumeSpecName "kube-api-access-2cjjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:20.045363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:20.045281    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cjjz\" (UniqueName: \"kubernetes.io/projected/410f3d85-4a38-40fc-b478-85e06686f9b6-kube-api-access-2cjjz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:20.045363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:20.045316    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/410f3d85-4a38-40fc-b478-85e06686f9b6-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:20.732092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:20.732066    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"
Oct 30 21:35:20.732092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:20.732085    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x" event={"ID":"410f3d85-4a38-40fc-b478-85e06686f9b6","Type":"ContainerDied","Data":"c1fdf7718842422d29c3b2765b17920545053edba1dc2104aa677d55a5f64782"}
Oct 30 21:35:20.732312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:20.732135    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1fdf7718842422d29c3b2765b17920545053edba1dc2104aa677d55a5f64782"
Oct 30 21:35:23.743021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.742986    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"]
Oct 30 21:35:23.744134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.744109    4918 cpu_manager.go:414] "containerMap: removing container" podUID="410f3d85-4a38-40fc-b478-85e06686f9b6" containerName="revoke-keys"
Oct 30 21:35:23.744134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.744133    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="410f3d85-4a38-40fc-b478-85e06686f9b6" containerName="revoke-keys"
Oct 30 21:35:23.744349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.744334    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="410f3d85-4a38-40fc-b478-85e06686f9b6" containerName="revoke-keys"
Oct 30 21:35:23.748596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.748575    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:23.772231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.772198    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"]
Oct 30 21:35:23.789550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.789513    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:23.789754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.789591    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:23.789754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.789739    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85d2d\" (UniqueName: \"kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-kube-api-access-85d2d\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:23.789905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.789880    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:23.891217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.891184    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:23.891436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.891311    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:23.891436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.891345    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:23.891588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.891475    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85d2d\" (UniqueName: \"kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-kube-api-access-85d2d\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:23.892003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.891979    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:23.892100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.892024    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:23.894155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.894134    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:23.902405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:23.902369    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85d2d\" (UniqueName: \"kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-kube-api-access-85d2d\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:24.079563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:24.079470    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:24.389479 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:35:24.389447    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95391f04_5961_4458_becd_19f29bebbc42.slice/crio-b01282305162c1d280f3336a6d70088b214775e5cdaddf618dd23ef03e8f41bb WatchSource:0}: Error finding container b01282305162c1d280f3336a6d70088b214775e5cdaddf618dd23ef03e8f41bb: Status 404 returned error can't find the container with id b01282305162c1d280f3336a6d70088b214775e5cdaddf618dd23ef03e8f41bb
Oct 30 21:35:24.389791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:24.389769    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"]
Oct 30 21:35:24.769242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:24.769207    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj" event={"ID":"95391f04-5961-4458-becd-19f29bebbc42","Type":"ContainerStarted","Data":"a1b6a74c4983b57c5a5a5b8b13ac4b37b49d9bfa575e87c19c021edd68f5b18b"}
Oct 30 21:35:24.769671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:24.769269    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj" event={"ID":"95391f04-5961-4458-becd-19f29bebbc42","Type":"ContainerStarted","Data":"b01282305162c1d280f3336a6d70088b214775e5cdaddf618dd23ef03e8f41bb"}
Oct 30 21:35:24.769671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:24.769294    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:24.795958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:24.795912    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj" podStartSLOduration=1.795899938 podStartE2EDuration="1.795899938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:24.795788622 +0000 UTC m=+2037.842347785" watchObservedRunningTime="2026-10-30 21:35:24.795899938 +0000 UTC m=+2037.842459083"
Oct 30 21:35:27.711879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:27.711839    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"]
Oct 30 21:35:29.163212 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:35:29.163175    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:35:29.726166 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:29.726131    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:35:32.712507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:32.712464    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj" podUID="26d4dd38-bf08-4b1d-bd2a-ece135534a55" containerName="payload-processing" containerID="cri-o://e068ea1becb5a9ebb8bc739e53f73c7a6de4df42deb7ab6d969ff15de0428ef2" gracePeriod=25
Oct 30 21:35:32.853822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:32.853777    4918 generic.go:356] "Generic (PLEG): container finished" podID="26d4dd38-bf08-4b1d-bd2a-ece135534a55" containerID="e068ea1becb5a9ebb8bc739e53f73c7a6de4df42deb7ab6d969ff15de0428ef2" exitCode=0
Oct 30 21:35:32.854038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:32.853852    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj" event={"ID":"26d4dd38-bf08-4b1d-bd2a-ece135534a55","Type":"ContainerDied","Data":"e068ea1becb5a9ebb8bc739e53f73c7a6de4df42deb7ab6d969ff15de0428ef2"}
Oct 30 21:35:33.093974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.093951    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:35:33.258425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.258379    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-routing-overlay-volume\") pod \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") "
Oct 30 21:35:33.258633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.258452    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-plugins-config-volume\") pod \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") "
Oct 30 21:35:33.258633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.258541    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-kube-api-access-5bh6l\" (UniqueName: \"kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-kube-api-access-5bh6l\") pod \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") "
Oct 30 21:35:33.258755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.258729    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-provider-credentials\" (UniqueName: \"kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-provider-credentials\") pod \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\" (UID: \"26d4dd38-bf08-4b1d-bd2a-ece135534a55\") "
Oct 30 21:35:33.258822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.258767    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-routing-overlay-volume" pod "26d4dd38-bf08-4b1d-bd2a-ece135534a55" (UID: "26d4dd38-bf08-4b1d-bd2a-ece135534a55"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:33.258875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.258821    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-plugins-config-volume" pod "26d4dd38-bf08-4b1d-bd2a-ece135534a55" (UID: "26d4dd38-bf08-4b1d-bd2a-ece135534a55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:33.259329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.259308    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:33.259471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.259332    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26d4dd38-bf08-4b1d-bd2a-ece135534a55-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:33.260886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.260867    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-kube-api-access-5bh6l" pod "26d4dd38-bf08-4b1d-bd2a-ece135534a55" (UID: "26d4dd38-bf08-4b1d-bd2a-ece135534a55"). InnerVolumeSpecName "kube-api-access-5bh6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:33.260983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.260966    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-provider-credentials" pod "26d4dd38-bf08-4b1d-bd2a-ece135534a55" (UID: "26d4dd38-bf08-4b1d-bd2a-ece135534a55"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:33.361664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.361627    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bh6l\" (UniqueName: \"kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-kube-api-access-5bh6l\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:33.361664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.361661    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/26d4dd38-bf08-4b1d-bd2a-ece135534a55-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:33.863915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.863880    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj" event={"ID":"26d4dd38-bf08-4b1d-bd2a-ece135534a55","Type":"ContainerDied","Data":"91c918b36ad4399c0ef273feff76161b1586fced34b6910491cb2dc005a41c19"}
Oct 30 21:35:33.864373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.863950    4918 scope.go:122] "RemoveContainer" containerID="e068ea1becb5a9ebb8bc739e53f73c7a6de4df42deb7ab6d969ff15de0428ef2"
Oct 30 21:35:33.864373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.863889    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"
Oct 30 21:35:33.895743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.895718    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"]
Oct 30 21:35:33.908132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:33.908112    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-7b5a7c/payload-processing-external-model-e2e-extmod-b-7b5a7c-56d4jwblj"]
Oct 30 21:35:35.370403 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:35.370357    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d4dd38-bf08-4b1d-bd2a-ece135534a55" path="/var/lib/kubelet/pods/26d4dd38-bf08-4b1d-bd2a-ece135534a55/volumes"
Oct 30 21:35:35.660120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:35.660031    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:35:35.709556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:35.709514    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:35:35.833653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:35.833618    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:36.031973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:36.031932    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"]
Oct 30 21:35:39.407144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.407108    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"]
Oct 30 21:35:39.412551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.412527    4918 cpu_manager.go:414] "containerMap: removing container" podUID="26d4dd38-bf08-4b1d-bd2a-ece135534a55" containerName="payload-processing"
Oct 30 21:35:39.412726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.412648    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d4dd38-bf08-4b1d-bd2a-ece135534a55" containerName="payload-processing"
Oct 30 21:35:39.412967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.412948    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d4dd38-bf08-4b1d-bd2a-ece135534a55" containerName="payload-processing"
Oct 30 21:35:39.418262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.418243    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.430195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.430132    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"]
Oct 30 21:35:39.501417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.484864    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.501417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.484965    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgjj5\" (UniqueName: \"kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-kube-api-access-cgjj5\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.501417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.485057    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-envoy\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.501417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.485083    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-podinfo\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.501417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.485154    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-credential-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.501417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.485315    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-certs\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.501417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.485354    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dede67-e6f7-4577-a8c6-8c4c671814d1-istiod-ca-cert\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.501417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.485474    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-token\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.501417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.485515    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-data\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.591595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.591554    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-envoy\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.591856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.591837    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-podinfo\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.592045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.592028    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-credential-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.592352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.592309    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-certs\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.592557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.592539    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dede67-e6f7-4577-a8c6-8c4c671814d1-istiod-ca-cert\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.592730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.592715    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-token\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.606527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.606488    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-data\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.606836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.606814    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.607044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.607027    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgjj5\" (UniqueName: \"kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-kube-api-access-cgjj5\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.609428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.608460    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.609428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.608979    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-credential-socket\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.609968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.609919    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-certs\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.610407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.610368    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-data\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.611184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.611163    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dede67-e6f7-4577-a8c6-8c4c671814d1-istiod-ca-cert\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.613567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.613545    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-envoy\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.616000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.615972    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-podinfo\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.622496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.621665    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-token\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.622496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.622013    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgjj5\" (UniqueName: \"kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-kube-api-access-cgjj5\") pod \"e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") " pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:39.744997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:39.744967    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:40.119318 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:35:40.119287    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dede67_e6f7_4577_a8c6_8c4c671814d1.slice/crio-8d50e89bc91f7fe7fbe574a4da3bc6f6fbdbfee28cf99a3c478c5ff500834c94 WatchSource:0}: Error finding container 8d50e89bc91f7fe7fbe574a4da3bc6f6fbdbfee28cf99a3c478c5ff500834c94: Status 404 returned error can't find the container with id 8d50e89bc91f7fe7fbe574a4da3bc6f6fbdbfee28cf99a3c478c5ff500834c94
Oct 30 21:35:40.121748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:40.121720    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"]
Oct 30 21:35:40.122871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:40.122840    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:35:40.122930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:40.122905    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:35:40.122971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:40.122934    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:35:41.032578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:41.032515    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw" podUID="35743d11-427d-46dd-a9d5-fa08373a702b" containerName="payload-processing" containerID="cri-o://b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00" gracePeriod=25
Oct 30 21:35:41.498226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:41.498177    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" event={"ID":"22dede67-e6f7-4577-a8c6-8c4c671814d1","Type":"ContainerStarted","Data":"88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079"}
Oct 30 21:35:41.499134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:41.499110    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" event={"ID":"22dede67-e6f7-4577-a8c6-8c4c671814d1","Type":"ContainerStarted","Data":"8d50e89bc91f7fe7fbe574a4da3bc6f6fbdbfee28cf99a3c478c5ff500834c94"}
Oct 30 21:35:41.567223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:41.567165    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" podStartSLOduration=2.567147435 podStartE2EDuration="2.567147435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:41.522893971 +0000 UTC m=+2054.569453128" watchObservedRunningTime="2026-10-30 21:35:41.567147435 +0000 UTC m=+2054.613706581"
Oct 30 21:35:41.750076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:41.749998    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:41.759982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:41.759588    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 21:35:41.759982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:41.759663    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 21:35:41.830470 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:35:41.828603    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35743d11_427d_46dd_a9d5_fa08373a702b.slice/crio-b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35743d11_427d_46dd_a9d5_fa08373a702b.slice/crio-conmon-b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:35:42.041564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.041486    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:35:42.143098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.143059    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-routing-overlay-volume\") pod \"35743d11-427d-46dd-a9d5-fa08373a702b\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") "
Oct 30 21:35:42.143268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.143127    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-plugins-config-volume\") pod \"35743d11-427d-46dd-a9d5-fa08373a702b\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") "
Oct 30 21:35:42.143385 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.143363    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-kube-api-access-tcm4p\" (UniqueName: \"kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-kube-api-access-tcm4p\") pod \"35743d11-427d-46dd-a9d5-fa08373a702b\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") "
Oct 30 21:35:42.143516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.143496    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-provider-credentials\") pod \"35743d11-427d-46dd-a9d5-fa08373a702b\" (UID: \"35743d11-427d-46dd-a9d5-fa08373a702b\") "
Oct 30 21:35:42.143864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.143490    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-routing-overlay-volume" pod "35743d11-427d-46dd-a9d5-fa08373a702b" (UID: "35743d11-427d-46dd-a9d5-fa08373a702b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:42.144109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.143759    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-plugins-config-volume" pod "35743d11-427d-46dd-a9d5-fa08373a702b" (UID: "35743d11-427d-46dd-a9d5-fa08373a702b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:42.144507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.144485    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:42.144602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.144510    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35743d11-427d-46dd-a9d5-fa08373a702b-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:42.146947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.146927    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-provider-credentials" pod "35743d11-427d-46dd-a9d5-fa08373a702b" (UID: "35743d11-427d-46dd-a9d5-fa08373a702b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:42.147056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.146972    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-kube-api-access-tcm4p" pod "35743d11-427d-46dd-a9d5-fa08373a702b" (UID: "35743d11-427d-46dd-a9d5-fa08373a702b"). InnerVolumeSpecName "kube-api-access-tcm4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:42.247508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.247464    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcm4p\" (UniqueName: \"kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-kube-api-access-tcm4p\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:42.247508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.247502    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35743d11-427d-46dd-a9d5-fa08373a702b-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:42.428313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.428224    4918 generic.go:356] "Generic (PLEG): container finished" podID="35743d11-427d-46dd-a9d5-fa08373a702b" containerID="b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00" exitCode=0
Oct 30 21:35:42.428313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.428269    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"
Oct 30 21:35:42.428590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.428306    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw" event={"ID":"35743d11-427d-46dd-a9d5-fa08373a702b","Type":"ContainerDied","Data":"b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00"}
Oct 30 21:35:42.428590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.428367    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw" event={"ID":"35743d11-427d-46dd-a9d5-fa08373a702b","Type":"ContainerDied","Data":"8170339598c7e42b9aa436bdbd26205a3100ad179f0b2be8ed2b2283ae3e1763"}
Oct 30 21:35:42.428590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.428422    4918 scope.go:122] "RemoveContainer" containerID="b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00"
Oct 30 21:35:42.441185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.441162    4918 scope.go:122] "RemoveContainer" containerID="b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00"
Oct 30 21:35:42.441510 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:35:42.441491    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00\": container with ID starting with b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00 not found: ID does not exist" containerID="b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00"
Oct 30 21:35:42.441599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.441522    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00"} err="failed to get container status \"b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00\": rpc error: code = NotFound desc = could not find container \"b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00\": container with ID starting with b2e85597a3f29272e18be043c6118433437ecc591f59d794f3d00da8261cfc00 not found: ID does not exist"
Oct 30 21:35:42.521469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.521441    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"]
Oct 30 21:35:42.526098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.526076    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-5c75qwjcw"]
Oct 30 21:35:42.800417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.799966    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 21:35:42.800417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:42.800048    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 21:35:43.145141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.145064    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35743d11-427d-46dd-a9d5-fa08373a702b" path="/var/lib/kubelet/pods/35743d11-427d-46dd-a9d5-fa08373a702b/volumes"
Oct 30 21:35:43.557228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.554849    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"]
Oct 30 21:35:43.557228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.556262    4918 cpu_manager.go:414] "containerMap: removing container" podUID="35743d11-427d-46dd-a9d5-fa08373a702b" containerName="payload-processing"
Oct 30 21:35:43.557228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.556289    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35743d11-427d-46dd-a9d5-fa08373a702b" containerName="payload-processing"
Oct 30 21:35:43.557228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.556553    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="35743d11-427d-46dd-a9d5-fa08373a702b" containerName="payload-processing"
Oct 30 21:35:43.656435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.656172    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:43.677414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.668780    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"agc-cred-2f614df36ac6\""
Oct 30 21:35:43.677414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.668921    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"kube-root-ca.crt\""
Oct 30 21:35:43.677414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.668995    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"routing-overlay\""
Oct 30 21:35:43.677414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.669077    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"openshift-service-ca.crt\""
Oct 30 21:35:43.677414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.669140    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"payload-processing-external-model-plugins-e2e-extmod-b-36a099\""
Oct 30 21:35:43.677414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.669233    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"payload-processing-external-model-e2e--3e617b48-dockercfg-d4sw2\""
Oct 30 21:35:43.786413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.758895    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trfdw\" (UniqueName: \"kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-kube-api-access-trfdw\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:43.786413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.758971    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:43.786413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.759155    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:43.786413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.759230    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:43.786413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.759482    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 21:35:43.786413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.759531    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 21:35:43.818521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.818429    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"]
Oct 30 21:35:43.874379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.874345    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:43.875410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.875366    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:43.875874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.875850    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trfdw\" (UniqueName: \"kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-kube-api-access-trfdw\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:43.876050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.876034    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:43.877428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.877406    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:43.878280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.878258    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:43.932730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.931655    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:43.957125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:43.957085    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trfdw\" (UniqueName: \"kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-kube-api-access-trfdw\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:44.208922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:44.208840    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:44.630211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:44.629907    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"]
Oct 30 21:35:44.746509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:44.746477    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 21:35:44.746680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:44.746539    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 21:35:45.650788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:45.650754    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc" event={"ID":"87c78554-01ac-48cc-9998-c8471e310d45","Type":"ContainerStarted","Data":"57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73"}
Oct 30 21:35:45.651228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:45.650809    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc" event={"ID":"87c78554-01ac-48cc-9998-c8471e310d45","Type":"ContainerStarted","Data":"7173b64d1f353ac8f611b423ae0dbc6876e6bc2cc70043c4e4af2f74e0f3a6fb"}
Oct 30 21:35:45.651228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:45.650848    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:45.682242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:45.682180    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc" podStartSLOduration=2.682166193 podStartE2EDuration="2.682166193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:45.674298181 +0000 UTC m=+2058.720857336" watchObservedRunningTime="2026-10-30 21:35:45.682166193 +0000 UTC m=+2058.728725338"
Oct 30 21:35:45.746426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:45.746373    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 21:35:45.746602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:45.746445    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 21:35:46.005416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:45.995213    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"]
Oct 30 21:35:46.005416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.002584    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.015416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.008021    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7b5a7c-openshift-default-dockercfg-ftslg\""
Oct 30 21:35:46.033196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.033164    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"]
Oct 30 21:35:46.067457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.067373    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-data\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.067647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.067534    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-credential-socket\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.067730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.067702    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-envoy\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.067892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.067872    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-certs\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.067966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.067921    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ck4\" (UniqueName: \"kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-kube-api-access-s4ck4\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.068027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.068010    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-podinfo\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.068087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.068056    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-socket\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.068150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.068124    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-token\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.068221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.068197    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30e391e-32ba-492a-98af-ac3c4d6089d1-istiod-ca-cert\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.170683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.170637    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-data\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.170868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.170766    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-credential-socket\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.170868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.170837    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-envoy\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.170956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.170947    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-certs\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.170993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.170985    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ck4\" (UniqueName: \"kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-kube-api-access-s4ck4\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.171070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.171045    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-podinfo\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.171259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.171086    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-socket\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.171259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.171094    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-data\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.171259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.171171    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-token\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.171259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.171250    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30e391e-32ba-492a-98af-ac3c4d6089d1-istiod-ca-cert\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.171576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.171303    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-certs\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.171576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.171415    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-socket\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.171576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.171512    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-credential-socket\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.172054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.172031    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30e391e-32ba-492a-98af-ac3c4d6089d1-istiod-ca-cert\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.173783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.173755    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-envoy\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.174187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.174164    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-podinfo\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.182909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.182882    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ck4\" (UniqueName: \"kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-kube-api-access-s4ck4\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.183013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.182885    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-token\") pod \"e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") " pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.380656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.380570    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:46.749419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.748880    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 21:35:46.749419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:46.748942    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 21:35:47.527344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:47.527315    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"]
Oct 30 21:35:47.528923 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:35:47.528893    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30e391e_32ba_492a_98af_ac3c4d6089d1.slice/crio-6c10569304d6d4493b060b8cc6fef79da7ac1bf844099f10b209f79c5a693afb WatchSource:0}: Error finding container 6c10569304d6d4493b060b8cc6fef79da7ac1bf844099f10b209f79c5a693afb: Status 404 returned error can't find the container with id 6c10569304d6d4493b060b8cc6fef79da7ac1bf844099f10b209f79c5a693afb
Oct 30 21:35:47.532358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:47.532319    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:35:47.532516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:47.532431    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:35:47.532516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:47.532483    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:35:47.748824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:47.748789    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 21:35:47.749016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:47.748847    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 21:35:47.772663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:47.772627    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" event={"ID":"f30e391e-32ba-492a-98af-ac3c4d6089d1","Type":"ContainerStarted","Data":"2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf"}
Oct 30 21:35:47.773236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:47.772690    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" event={"ID":"f30e391e-32ba-492a-98af-ac3c4d6089d1","Type":"ContainerStarted","Data":"6c10569304d6d4493b060b8cc6fef79da7ac1bf844099f10b209f79c5a693afb"}
Oct 30 21:35:47.821596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:47.821484    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" podStartSLOduration=2.821462288 podStartE2EDuration="2.821462288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:47.798656346 +0000 UTC m=+2060.845215516" watchObservedRunningTime="2026-10-30 21:35:47.821462288 +0000 UTC m=+2060.868021432"
Oct 30 21:35:47.973091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:47.973046    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:35:47.973295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:47.973107    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:35:48.431498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:48.431417    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:48.459089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:48.459057    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 21:35:48.459424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:48.459366    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 21:35:48.746755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:48.746714    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 21:35:48.746937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:48.746777    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 21:35:49.519420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:49.513524    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 21:35:49.519420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:49.513583    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 21:35:50.266765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:50.266726    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:50.267669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:50.267645    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:50.268475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:50.268455    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:35:50.382487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:50.382088    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 21:35:50.382487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:50.382148    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 21:35:51.511456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:51.511416    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 21:35:51.514316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:51.514287    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 21:35:51.514880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:51.512325    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:35:52.383154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:52.383112    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 21:35:52.383768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:52.383736    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 21:35:53.082821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:53.082740    4918 generic.go:356] "Generic (PLEG): container finished" podID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerID="51dc0ec76e9d0e56db1226c4800fe338cc898383a972123c56422222bab05250" exitCode=0
Oct 30 21:35:53.082821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:53.082810    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" event={"ID":"be5b070d-b6a5-4eb0-be88-300467b249f9","Type":"ContainerDied","Data":"51dc0ec76e9d0e56db1226c4800fe338cc898383a972123c56422222bab05250"}
Oct 30 21:35:53.083289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:53.083230    4918 scope.go:122] "RemoveContainer" containerID="51dc0ec76e9d0e56db1226c4800fe338cc898383a972123c56422222bab05250"
Oct 30 21:35:53.085212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:53.085186    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:35:53.085294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:53.085252    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:35:53.085294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:53.085287    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:35:53.381617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:53.381532    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 21:35:53.381617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:53.381603    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 21:35:54.081581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:54.080904    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"]
Oct 30 21:35:54.150639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:54.150591    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" event={"ID":"be5b070d-b6a5-4eb0-be88-300467b249f9","Type":"ContainerStarted","Data":"8b4949621a2fe48fb367cdd8011c4509231cfbf111c108cde3751c5482a8260a"}
Oct 30 21:35:54.230908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:54.230877    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:35:54.249839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:54.249807    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 21:35:54.257531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:54.257476    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 21:35:54.393626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:54.393530    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 21:35:54.393937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:54.393902    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 21:35:55.270419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:55.230750    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 21:35:55.270419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:55.230907    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 21:35:55.388204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:55.386036    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 21:35:55.388204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:55.386098    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 21:35:56.231708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:56.231640    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 21:35:56.232008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:56.231975    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 21:35:56.382998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:56.382966    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Oct 30 21:35:56.383623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:56.383592    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Oct 30 21:35:56.759597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:56.759571    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:35:57.231180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:57.231148    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 21:35:57.231369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:57.231211    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 21:35:57.486475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:57.479711    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:57.486475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:57.481503    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:57.526466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:57.526434    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:35:58.236576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:58.236525    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 21:35:58.236869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:58.236836    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 21:35:59.082423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.081495    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj" podUID="95391f04-5961-4458-becd-19f29bebbc42" containerName="payload-processing" containerID="cri-o://a1b6a74c4983b57c5a5a5b8b13ac4b37b49d9bfa575e87c19c021edd68f5b18b" gracePeriod=25
Oct 30 21:35:59.231099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.231059    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body=
Oct 30 21:35:59.232175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.232140    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused"
Oct 30 21:35:59.254003 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:35:59.253977    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:35:59.368659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.368613    4918 generic.go:356] "Generic (PLEG): container finished" podID="95391f04-5961-4458-becd-19f29bebbc42" containerID="a1b6a74c4983b57c5a5a5b8b13ac4b37b49d9bfa575e87c19c021edd68f5b18b" exitCode=0
Oct 30 21:35:59.369013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.368907    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj" event={"ID":"95391f04-5961-4458-becd-19f29bebbc42","Type":"ContainerDied","Data":"a1b6a74c4983b57c5a5a5b8b13ac4b37b49d9bfa575e87c19c021edd68f5b18b"}
Oct 30 21:35:59.481474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.481449    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:35:59.616554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.616511    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-kube-api-access-85d2d\" (UniqueName: \"kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-kube-api-access-85d2d\") pod \"95391f04-5961-4458-becd-19f29bebbc42\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") "
Oct 30 21:35:59.616753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.616722    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-plugins-config-volume\") pod \"95391f04-5961-4458-becd-19f29bebbc42\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") "
Oct 30 21:35:59.616891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.616772    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-routing-overlay-volume\") pod \"95391f04-5961-4458-becd-19f29bebbc42\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") "
Oct 30 21:35:59.616956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.616908    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-provider-credentials\" (UniqueName: \"kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-provider-credentials\") pod \"95391f04-5961-4458-becd-19f29bebbc42\" (UID: \"95391f04-5961-4458-becd-19f29bebbc42\") "
Oct 30 21:35:59.617130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.617107    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-plugins-config-volume" pod "95391f04-5961-4458-becd-19f29bebbc42" (UID: "95391f04-5961-4458-becd-19f29bebbc42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:59.617207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.617130    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-routing-overlay-volume" pod "95391f04-5961-4458-becd-19f29bebbc42" (UID: "95391f04-5961-4458-becd-19f29bebbc42"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:35:59.617616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.617596    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:59.617616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.617616    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95391f04-5961-4458-becd-19f29bebbc42-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:59.618913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.618868    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-kube-api-access-85d2d" pod "95391f04-5961-4458-becd-19f29bebbc42" (UID: "95391f04-5961-4458-becd-19f29bebbc42"). InnerVolumeSpecName "kube-api-access-85d2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:59.619269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.619247    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-provider-credentials" pod "95391f04-5961-4458-becd-19f29bebbc42" (UID: "95391f04-5961-4458-becd-19f29bebbc42"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:35:59.719582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.719544    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85d2d\" (UniqueName: \"kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-kube-api-access-85d2d\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:35:59.719582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:35:59.719579    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95391f04-5961-4458-becd-19f29bebbc42-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:36:00.255922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:00.255886    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:36:00.256539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:00.256518    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:36:00.259066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:00.259044    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:36:00.381742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:00.381712    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"
Oct 30 21:36:00.381742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:00.381728    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj" event={"ID":"95391f04-5961-4458-becd-19f29bebbc42","Type":"ContainerDied","Data":"b01282305162c1d280f3336a6d70088b214775e5cdaddf618dd23ef03e8f41bb"}
Oct 30 21:36:00.382706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:00.381803    4918 scope.go:122] "RemoveContainer" containerID="a1b6a74c4983b57c5a5a5b8b13ac4b37b49d9bfa575e87c19c021edd68f5b18b"
Oct 30 21:36:00.412896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:00.412867    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"]
Oct 30 21:36:00.417625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:00.417604    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-f49dtrbvj"]
Oct 30 21:36:01.080142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:01.080106    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95391f04-5961-4458-becd-19f29bebbc42" path="/var/lib/kubelet/pods/95391f04-5961-4458-becd-19f29bebbc42/volumes"
Oct 30 21:36:11.481929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:11.481750    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded"
Oct 30 21:36:11.670761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:11.670716    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:36:11.671355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:11.671325    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:36:11.674093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:11.674062    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:36:11.724627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:11.724571    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:36:11.820628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:11.820520    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:36:11.917745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:11.917698    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:36:11.937780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:11.937733    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:36:11.955809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:11.955766    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:36:11.973741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:11.973697    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:36:12.081072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:12.080977    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:36:12.113526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:12.082765    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:36:12.678419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:12.674463    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:36:12.678419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:12.675017    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:36:12.678419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:12.676893    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:36:12.678419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:12.677520    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:36:12.740564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:12.740522    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:36:12.741477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:12.741456    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:36:12.851893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:12.851865    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:36:12.861267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:12.861239    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:36:12.947362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:12.926146    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:36:17.093782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:17.093754    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"]
Oct 30 21:36:17.094201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:17.093791    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9529bf3e-kknjm"]
Oct 30 21:36:18.111961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:18.108823    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"]
Oct 30 21:36:18.119974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:18.119940    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83668192-ck8fs"]
Oct 30 21:36:19.079964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:19.079928    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc9f470-5c19-40fc-a15a-dcc783628049" path="/var/lib/kubelet/pods/dfc9f470-5c19-40fc-a15a-dcc783628049/volumes"
Oct 30 21:36:19.080525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:19.080509    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d" path="/var/lib/kubelet/pods/f613b20d-5ebd-4dcb-9a52-b08ee1e61e7d/volumes"
Oct 30 21:36:27.172079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:27.172046    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"]
Oct 30 21:36:27.175580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:27.175555    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71c06b8c-229h5"]
Oct 30 21:36:28.001307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:28.001276    4918 generic.go:356] "Generic (PLEG): container finished" podID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerID="a5f85956238ef40fa02cf7d14786d6184a719746e22d129a6dc239b3bed068a4" exitCode=0
Oct 30 21:36:28.001582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:28.001354    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" event={"ID":"fc03a02c-5042-4f3c-be33-4443682c2f6d","Type":"ContainerDied","Data":"a5f85956238ef40fa02cf7d14786d6184a719746e22d129a6dc239b3bed068a4"}
Oct 30 21:36:28.001888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:28.001871    4918 scope.go:122] "RemoveContainer" containerID="a5f85956238ef40fa02cf7d14786d6184a719746e22d129a6dc239b3bed068a4"
Oct 30 21:36:28.003813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:28.003783    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:36:28.003922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:28.003866    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:36:28.003922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:28.003917    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:36:29.022684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:29.022650    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" event={"ID":"fc03a02c-5042-4f3c-be33-4443682c2f6d","Type":"ContainerStarted","Data":"d633a872e2055586db2c1ec84f203ae6ff8cce330bed76aa9569cf9b66765834"}
Oct 30 21:36:29.069331 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:29.069299    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"]
Oct 30 21:36:29.082256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:29.082221    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709" path="/var/lib/kubelet/pods/665e9bcb-8c55-4ba9-bdc1-7dc9d4a3a709/volumes"
Oct 30 21:36:29.082678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:29.082658    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a24e1e6-5tfm9"]
Oct 30 21:36:29.162978 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:36:29.162945    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:36:29.926820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:29.926791    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:36:29.927016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:29.926972    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 21:36:29.927069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:29.927028    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 21:36:30.120330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:30.120286    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"]
Oct 30 21:36:30.280432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:30.280359    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:36:30.281982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:30.281957    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:36:30.927762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:30.927732    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 21:36:30.927953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:30.927782    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 21:36:31.081995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:31.081964    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1dbc5a-6e0e-4647-93c2-cce9a11cc705" path="/var/lib/kubelet/pods/4b1dbc5a-6e0e-4647-93c2-cce9a11cc705/volumes"
Oct 30 21:36:31.926766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:31.926721    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 21:36:31.927203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:31.926789    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 21:36:32.927002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:32.926963    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 21:36:32.927442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:32.927036    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 21:36:33.927149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:33.927118    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body=
Oct 30 21:36:33.927616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:33.927180    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused"
Oct 30 21:36:34.933553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:34.933523    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:36:34.934053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:34.933934    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:36:34.934538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:34.934521    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:36:35.121427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.121372    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d" podUID="0aed2990-6360-46eb-8322-07c97e074d2a" containerName="payload-processing" containerID="cri-o://65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257" gracePeriod=25
Oct 30 21:36:35.455906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.455881    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:36:35.511454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.511339    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-plugins-config-volume\") pod \"0aed2990-6360-46eb-8322-07c97e074d2a\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") "
Oct 30 21:36:35.511454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.511446    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-kube-api-access-54k89\" (UniqueName: \"kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-kube-api-access-54k89\") pod \"0aed2990-6360-46eb-8322-07c97e074d2a\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") "
Oct 30 21:36:35.511691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.511610    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-routing-overlay-volume\") pod \"0aed2990-6360-46eb-8322-07c97e074d2a\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") "
Oct 30 21:36:35.511691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.511655    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-provider-credentials\") pod \"0aed2990-6360-46eb-8322-07c97e074d2a\" (UID: \"0aed2990-6360-46eb-8322-07c97e074d2a\") "
Oct 30 21:36:35.511819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.511785    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-plugins-config-volume" pod "0aed2990-6360-46eb-8322-07c97e074d2a" (UID: "0aed2990-6360-46eb-8322-07c97e074d2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:35.511995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.511974    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-routing-overlay-volume" pod "0aed2990-6360-46eb-8322-07c97e074d2a" (UID: "0aed2990-6360-46eb-8322-07c97e074d2a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:35.512520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.512496    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:36:35.512653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.512636    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aed2990-6360-46eb-8322-07c97e074d2a-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:36:35.514238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.514212    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-provider-credentials" pod "0aed2990-6360-46eb-8322-07c97e074d2a" (UID: "0aed2990-6360-46eb-8322-07c97e074d2a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:35.514450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.514379    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-kube-api-access-54k89" pod "0aed2990-6360-46eb-8322-07c97e074d2a" (UID: "0aed2990-6360-46eb-8322-07c97e074d2a"). InnerVolumeSpecName "kube-api-access-54k89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:35.618044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.618011    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54k89\" (UniqueName: \"kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-kube-api-access-54k89\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:36:35.618044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:35.618040    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aed2990-6360-46eb-8322-07c97e074d2a-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:36:36.099665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:36.099633    4918 generic.go:356] "Generic (PLEG): container finished" podID="0aed2990-6360-46eb-8322-07c97e074d2a" containerID="65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257" exitCode=0
Oct 30 21:36:36.100172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:36.099675    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"
Oct 30 21:36:36.100172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:36.099713    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d" event={"ID":"0aed2990-6360-46eb-8322-07c97e074d2a","Type":"ContainerDied","Data":"65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257"}
Oct 30 21:36:36.100172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:36.099764    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d" event={"ID":"0aed2990-6360-46eb-8322-07c97e074d2a","Type":"ContainerDied","Data":"20bc909b5349ff3c956c95b28f35a93423805bf2c1be895f0c7adef5daffadf1"}
Oct 30 21:36:36.100172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:36.099786    4918 scope.go:122] "RemoveContainer" containerID="65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257"
Oct 30 21:36:36.112575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:36.112557    4918 scope.go:122] "RemoveContainer" containerID="65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257"
Oct 30 21:36:36.112850 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:36:36.112833    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257\": container with ID starting with 65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257 not found: ID does not exist" containerID="65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257"
Oct 30 21:36:36.112887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:36.112858    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257"} err="failed to get container status \"65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257\": rpc error: code = NotFound desc = could not find container \"65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257\": container with ID starting with 65538d4075aa6ffb7db0776ab3e1de03175a8cac29d6d603f84734cacc140257 not found: ID does not exist"
Oct 30 21:36:36.264150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:36.264068    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"]
Oct 30 21:36:36.272707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:36.272677    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ff7nj6d"]
Oct 30 21:36:37.159882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:37.159236    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aed2990-6360-46eb-8322-07c97e074d2a" path="/var/lib/kubelet/pods/0aed2990-6360-46eb-8322-07c97e074d2a/volumes"
Oct 30 21:36:46.562753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.562718    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"]
Oct 30 21:36:46.564741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.564701    4918 cpu_manager.go:414] "containerMap: removing container" podUID="0aed2990-6360-46eb-8322-07c97e074d2a" containerName="payload-processing"
Oct 30 21:36:46.564915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.564900    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aed2990-6360-46eb-8322-07c97e074d2a" containerName="payload-processing"
Oct 30 21:36:46.565033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.565022    4918 cpu_manager.go:414] "containerMap: removing container" podUID="95391f04-5961-4458-becd-19f29bebbc42" containerName="payload-processing"
Oct 30 21:36:46.565185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.565172    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95391f04-5961-4458-becd-19f29bebbc42" containerName="payload-processing"
Oct 30 21:36:46.565589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.565566    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="95391f04-5961-4458-becd-19f29bebbc42" containerName="payload-processing"
Oct 30 21:36:46.565710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.565614    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aed2990-6360-46eb-8322-07c97e074d2a" containerName="payload-processing"
Oct 30 21:36:46.574449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.572212    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:46.575752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.575727    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"]
Oct 30 21:36:46.576481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.576453    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:36:46.576595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.576569    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:36:46.577908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.577890    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:36:46.577997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.577940    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:36:46.747182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.747150    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-120178ce-xxd72\" (UID: \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:46.747430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.747403    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7lk8\" (UniqueName: \"kubernetes.io/projected/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-kube-api-access-b7lk8\") pod \"maas-api-revoke-sub-120178ce-xxd72\" (UID: \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:46.747515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.747495    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-120178ce-xxd72\" (UID: \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:46.852200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.851647    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7lk8\" (UniqueName: \"kubernetes.io/projected/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-kube-api-access-b7lk8\") pod \"maas-api-revoke-sub-120178ce-xxd72\" (UID: \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:46.852200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.851724    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-120178ce-xxd72\" (UID: \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:46.852200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.851976    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-120178ce-xxd72\" (UID: \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:46.859737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.859685    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-120178ce-xxd72\" (UID: \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:46.862168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.862138    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-120178ce-xxd72\" (UID: \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:46.867756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.867737    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7lk8\" (UniqueName: \"kubernetes.io/projected/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-kube-api-access-b7lk8\") pod \"maas-api-revoke-sub-120178ce-xxd72\" (UID: \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:46.900027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:46.900005    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:47.187435 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:36:47.187363    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f2f1b1_e81c_4ed1_a74e_79477e0551ea.slice/crio-718a3c386977eec8d3378eced288e205397808a241bc0fb81e73592f5465bb56 WatchSource:0}: Error finding container 718a3c386977eec8d3378eced288e205397808a241bc0fb81e73592f5465bb56: Status 404 returned error can't find the container with id 718a3c386977eec8d3378eced288e205397808a241bc0fb81e73592f5465bb56
Oct 30 21:36:47.188303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:47.188279    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"]
Oct 30 21:36:47.337879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:47.331546    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72" event={"ID":"64f2f1b1-e81c-4ed1-a74e-79477e0551ea","Type":"ContainerStarted","Data":"718a3c386977eec8d3378eced288e205397808a241bc0fb81e73592f5465bb56"}
Oct 30 21:36:48.342142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:48.342107    4918 generic.go:356] "Generic (PLEG): container finished" podID="64f2f1b1-e81c-4ed1-a74e-79477e0551ea" containerID="0b7c5a59d93e225188d4b9320d2bc2e2a4af062f60ad2546c9420d820a45246b" exitCode=0
Oct 30 21:36:48.342608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:48.342187    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72" event={"ID":"64f2f1b1-e81c-4ed1-a74e-79477e0551ea","Type":"ContainerDied","Data":"0b7c5a59d93e225188d4b9320d2bc2e2a4af062f60ad2546c9420d820a45246b"}
Oct 30 21:36:49.533593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:49.533567    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:49.613528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:49.613440    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-openshift-ca\") pod \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\" (UID: \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\") "
Oct 30 21:36:49.613712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:49.613613    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-xks-ca\") pod \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\" (UID: \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\") "
Oct 30 21:36:49.613712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:49.613662    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-kube-api-access-b7lk8\" (UniqueName: \"kubernetes.io/projected/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-kube-api-access-b7lk8\") pod \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\" (UID: \"64f2f1b1-e81c-4ed1-a74e-79477e0551ea\") "
Oct 30 21:36:49.613882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:49.613850    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-openshift-ca" pod "64f2f1b1-e81c-4ed1-a74e-79477e0551ea" (UID: "64f2f1b1-e81c-4ed1-a74e-79477e0551ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:36:49.614513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:49.614486    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:36:49.616080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:49.616050    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-xks-ca" pod "64f2f1b1-e81c-4ed1-a74e-79477e0551ea" (UID: "64f2f1b1-e81c-4ed1-a74e-79477e0551ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:36:49.616506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:49.616481    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-kube-api-access-b7lk8" pod "64f2f1b1-e81c-4ed1-a74e-79477e0551ea" (UID: "64f2f1b1-e81c-4ed1-a74e-79477e0551ea"). InnerVolumeSpecName "kube-api-access-b7lk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:36:49.716445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:49.716412    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:36:49.716445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:49.716442    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7lk8\" (UniqueName: \"kubernetes.io/projected/64f2f1b1-e81c-4ed1-a74e-79477e0551ea-kube-api-access-b7lk8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:36:50.241432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.241381    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"]
Oct 30 21:36:50.242965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.242490    4918 cpu_manager.go:414] "containerMap: removing container" podUID="64f2f1b1-e81c-4ed1-a74e-79477e0551ea" containerName="revoke-keys"
Oct 30 21:36:50.242965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.242513    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f2f1b1-e81c-4ed1-a74e-79477e0551ea" containerName="revoke-keys"
Oct 30 21:36:50.242965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.242760    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f2f1b1-e81c-4ed1-a74e-79477e0551ea" containerName="revoke-keys"
Oct 30 21:36:50.247721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.247693    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.288035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.287993    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"]
Oct 30 21:36:50.331351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.331317    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.331589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.331403    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.331658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.331610    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.331658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.331633    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.331762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.331704    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.331762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.331728    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.331882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.331773    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.331882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.331848    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq9cd\" (UniqueName: \"kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-kube-api-access-wq9cd\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.332009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.331899    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.363460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.363432    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72" event={"ID":"64f2f1b1-e81c-4ed1-a74e-79477e0551ea","Type":"ContainerDied","Data":"718a3c386977eec8d3378eced288e205397808a241bc0fb81e73592f5465bb56"}
Oct 30 21:36:50.363620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.363468    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"
Oct 30 21:36:50.363620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.363489    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="718a3c386977eec8d3378eced288e205397808a241bc0fb81e73592f5465bb56"
Oct 30 21:36:50.433818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.433781    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.433818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.433824    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.434036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.433863    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.434036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.433886    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.434036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.433910    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.434036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.433967    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq9cd\" (UniqueName: \"kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-kube-api-access-wq9cd\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.434270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.434053    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.434339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.434276    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.434339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.434325    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.434482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.434360    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.434703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.434679    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.434840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.434739    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.434840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.434750    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.434840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.434796    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.436416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.436384    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.436888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.436872    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.445031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.445008    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.445445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.445426    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq9cd\" (UniqueName: \"kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-kube-api-access-wq9cd\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.565426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.565319    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:50.857869 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:36:50.857834    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16b9f2a_0ed2_41ce_b956_a5d5d0081788.slice/crio-f896020e9b8979a5d095c9d98c7ce945a1aa65cd6939a43823f9d46bb399df3a WatchSource:0}: Error finding container f896020e9b8979a5d095c9d98c7ce945a1aa65cd6939a43823f9d46bb399df3a: Status 404 returned error can't find the container with id f896020e9b8979a5d095c9d98c7ce945a1aa65cd6939a43823f9d46bb399df3a
Oct 30 21:36:50.860299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.860270    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"]
Oct 30 21:36:50.861845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.861799    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:36:50.861962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.861892    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:36:50.861962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:50.861933    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:36:51.375685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:51.375652    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds" event={"ID":"a16b9f2a-0ed2-41ce-b956-a5d5d0081788","Type":"ContainerStarted","Data":"75839efcc128588612537b2a6031e36f90a132a2e08c88d857c80879f934be0b"}
Oct 30 21:36:51.375863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:51.375702    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds" event={"ID":"a16b9f2a-0ed2-41ce-b956-a5d5d0081788","Type":"ContainerStarted","Data":"f896020e9b8979a5d095c9d98c7ce945a1aa65cd6939a43823f9d46bb399df3a"}
Oct 30 21:36:51.410489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:51.410431    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds" podStartSLOduration=1.410410677 podStartE2EDuration="1.410410677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:51.402502249 +0000 UTC m=+2124.449061418" watchObservedRunningTime="2026-10-30 21:36:51.410410677 +0000 UTC m=+2124.456969816"
Oct 30 21:36:51.565587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:51.565557    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:52.567594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:52.567563    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 30 21:36:52.567979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:52.567614    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds" podUID="a16b9f2a-0ed2-41ce-b956-a5d5d0081788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 30 21:36:53.146638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:53.146506    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"]
Oct 30 21:36:53.146638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:53.146578    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cabdb7f4-49bjt"]
Oct 30 21:36:53.566334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:53.566302    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 30 21:36:53.566545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:53.566351    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds" podUID="a16b9f2a-0ed2-41ce-b956-a5d5d0081788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 30 21:36:54.566239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:54.566202    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 30 21:36:54.566742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:54.566267    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds" podUID="a16b9f2a-0ed2-41ce-b956-a5d5d0081788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 30 21:36:55.081235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:55.081204    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d7485b-dbf3-4b11-9973-b270bcaceb2e" path="/var/lib/kubelet/pods/c6d7485b-dbf3-4b11-9973-b270bcaceb2e/volumes"
Oct 30 21:36:55.568420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:55.567784    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 30 21:36:55.568420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:55.567872    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds" podUID="a16b9f2a-0ed2-41ce-b956-a5d5d0081788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 30 21:36:56.565988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:56.565954    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused" start-of-body=
Oct 30 21:36:56.566190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:56.566023    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds" podUID="a16b9f2a-0ed2-41ce-b956-a5d5d0081788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.56:15021/healthz/ready\": dial tcp 10.132.1.56:15021: connect: connection refused"
Oct 30 21:36:57.810513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.808138    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:57.810513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.809310    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:57.811659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.811605    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:36:57.863620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.863588    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"]
Oct 30 21:36:57.871456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.871428    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:57.879371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.879339    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"agc-cred-d390aee96ac0\""
Oct 30 21:36:57.879587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.879484    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"payload-processing-external-model-e2e--8e1d66cd-dockercfg-h75qf\""
Oct 30 21:36:57.879587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.879521    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"agc-cred-42957d623967\""
Oct 30 21:36:57.887452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.884651    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"openshift-service-ca.crt\""
Oct 30 21:36:57.887452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.884749    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea\""
Oct 30 21:36:57.887452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.884836    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"routing-overlay\""
Oct 30 21:36:57.887452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.884976    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"kube-root-ca.crt\""
Oct 30 21:36:57.887452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:57.885781    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"]
Oct 30 21:36:58.001010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.000978    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4jtk\" (UniqueName: \"kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-kube-api-access-b4jtk\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.001163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.001090    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.001271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.001257    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.001487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.001470    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.129468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.118691    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.129468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.118958    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.129468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.119016    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4jtk\" (UniqueName: \"kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-kube-api-access-b4jtk\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.129468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.119101    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.129468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.120050    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.134342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.134307    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.227852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.156173    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.228095 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.181475    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"]
Oct 30 21:36:58.234480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.186168    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4jtk\" (UniqueName: \"kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-kube-api-access-b4jtk\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.288666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.288627    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"]
Oct 30 21:36:58.289066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.289046    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:36:58.427473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.423911    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fe9de59-7wkrs\" (UID: \"fa0b785f-07ed-4382-af03-5709574a77bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:36:58.462687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.437759    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98q7b\" (UniqueName: \"kubernetes.io/projected/fa0b785f-07ed-4382-af03-5709574a77bd-kube-api-access-98q7b\") pod \"maas-api-revoke-sub-9fe9de59-7wkrs\" (UID: \"fa0b785f-07ed-4382-af03-5709574a77bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:36:58.462687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.437914    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fe9de59-7wkrs\" (UID: \"fa0b785f-07ed-4382-af03-5709574a77bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:36:58.462687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.452264    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:36:58.462687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.453100    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:36:58.462687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.453160    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:36:58.462687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.453214    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:36:58.530423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.522962    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:58.544428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.540929    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fe9de59-7wkrs\" (UID: \"fa0b785f-07ed-4382-af03-5709574a77bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:36:58.544428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.541036    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98q7b\" (UniqueName: \"kubernetes.io/projected/fa0b785f-07ed-4382-af03-5709574a77bd-kube-api-access-98q7b\") pod \"maas-api-revoke-sub-9fe9de59-7wkrs\" (UID: \"fa0b785f-07ed-4382-af03-5709574a77bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:36:58.544428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.541110    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fe9de59-7wkrs\" (UID: \"fa0b785f-07ed-4382-af03-5709574a77bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:36:58.545649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.545573    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fe9de59-7wkrs\" (UID: \"fa0b785f-07ed-4382-af03-5709574a77bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:36:58.556237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.555144    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fe9de59-7wkrs\" (UID: \"fa0b785f-07ed-4382-af03-5709574a77bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:36:58.579343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.579262    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98q7b\" (UniqueName: \"kubernetes.io/projected/fa0b785f-07ed-4382-af03-5709574a77bd-kube-api-access-98q7b\") pod \"maas-api-revoke-sub-9fe9de59-7wkrs\" (UID: \"fa0b785f-07ed-4382-af03-5709574a77bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:36:58.590934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.590905    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:36:58.899630 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:36:58.899602    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a3ed3c_4600_4ae7_95a1_cc183597cdd7.slice/crio-79f0fe3849c99ec78470106fc86ab04aa722d576f4473c2624f1dd2952e447d2 WatchSource:0}: Error finding container 79f0fe3849c99ec78470106fc86ab04aa722d576f4473c2624f1dd2952e447d2: Status 404 returned error can't find the container with id 79f0fe3849c99ec78470106fc86ab04aa722d576f4473c2624f1dd2952e447d2
Oct 30 21:36:58.900025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.899983    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"]
Oct 30 21:36:58.904947 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:36:58.904919    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0b785f_07ed_4382_af03_5709574a77bd.slice/crio-12de3cff82e89e741bfb35da8cab1729683831007d489e8a19e5f32b28fc5b45 WatchSource:0}: Error finding container 12de3cff82e89e741bfb35da8cab1729683831007d489e8a19e5f32b28fc5b45: Status 404 returned error can't find the container with id 12de3cff82e89e741bfb35da8cab1729683831007d489e8a19e5f32b28fc5b45
Oct 30 21:36:58.905302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:58.905280    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"]
Oct 30 21:36:59.163064 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:36:59.163029    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:36:59.607206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:59.607103    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs" event={"ID":"fa0b785f-07ed-4382-af03-5709574a77bd","Type":"ContainerStarted","Data":"1e15d4a6a5eb0155f7d61622394fa8173aca0c94789223b6efbaff0f53db8c41"}
Oct 30 21:36:59.607434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:59.607208    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs" event={"ID":"fa0b785f-07ed-4382-af03-5709574a77bd","Type":"ContainerStarted","Data":"12de3cff82e89e741bfb35da8cab1729683831007d489e8a19e5f32b28fc5b45"}
Oct 30 21:36:59.609613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:59.609583    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5" event={"ID":"89a3ed3c-4600-4ae7-95a1-cc183597cdd7","Type":"ContainerStarted","Data":"a4fe49267f08f9ce34a20b989be6bf90e7d5ad4eb46ef709b12541ed4971536a"}
Oct 30 21:36:59.609774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:59.609629    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5" event={"ID":"89a3ed3c-4600-4ae7-95a1-cc183597cdd7","Type":"ContainerStarted","Data":"79f0fe3849c99ec78470106fc86ab04aa722d576f4473c2624f1dd2952e447d2"}
Oct 30 21:36:59.609859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:59.609781    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:36:59.637113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:59.637062    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs" podStartSLOduration=1.637048281 podStartE2EDuration="1.637048281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:59.631530489 +0000 UTC m=+2132.678089661" watchObservedRunningTime="2026-10-30 21:36:59.637048281 +0000 UTC m=+2132.683607425"
Oct 30 21:36:59.664550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:36:59.664496    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5" podStartSLOduration=2.664481853 podStartE2EDuration="2.664481853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:59.657197681 +0000 UTC m=+2132.703757034" watchObservedRunningTime="2026-10-30 21:36:59.664481853 +0000 UTC m=+2132.711040999"
Oct 30 21:37:00.619805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:00.619767    4918 generic.go:356] "Generic (PLEG): container finished" podID="fa0b785f-07ed-4382-af03-5709574a77bd" containerID="1e15d4a6a5eb0155f7d61622394fa8173aca0c94789223b6efbaff0f53db8c41" exitCode=0
Oct 30 21:37:00.620201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:00.619855    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs" event={"ID":"fa0b785f-07ed-4382-af03-5709574a77bd","Type":"ContainerDied","Data":"1e15d4a6a5eb0155f7d61622394fa8173aca0c94789223b6efbaff0f53db8c41"}
Oct 30 21:37:01.910174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:01.910147    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:37:02.025995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:02.025967    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-openshift-ca\") pod \"fa0b785f-07ed-4382-af03-5709574a77bd\" (UID: \"fa0b785f-07ed-4382-af03-5709574a77bd\") "
Oct 30 21:37:02.026187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:02.026151    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0b785f-07ed-4382-af03-5709574a77bd-kube-api-access-98q7b\" (UniqueName: \"kubernetes.io/projected/fa0b785f-07ed-4382-af03-5709574a77bd-kube-api-access-98q7b\") pod \"fa0b785f-07ed-4382-af03-5709574a77bd\" (UID: \"fa0b785f-07ed-4382-af03-5709574a77bd\") "
Oct 30 21:37:02.026335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:02.026316    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-xks-ca\") pod \"fa0b785f-07ed-4382-af03-5709574a77bd\" (UID: \"fa0b785f-07ed-4382-af03-5709574a77bd\") "
Oct 30 21:37:02.026459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:02.026319    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-openshift-ca" pod "fa0b785f-07ed-4382-af03-5709574a77bd" (UID: "fa0b785f-07ed-4382-af03-5709574a77bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:02.027095 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:02.027071    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:02.028494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:02.028472    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0b785f-07ed-4382-af03-5709574a77bd-kube-api-access-98q7b" pod "fa0b785f-07ed-4382-af03-5709574a77bd" (UID: "fa0b785f-07ed-4382-af03-5709574a77bd"). InnerVolumeSpecName "kube-api-access-98q7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:02.028589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:02.028470    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-xks-ca" pod "fa0b785f-07ed-4382-af03-5709574a77bd" (UID: "fa0b785f-07ed-4382-af03-5709574a77bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:02.129249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:02.129205    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0b785f-07ed-4382-af03-5709574a77bd-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:02.129249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:02.129233    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98q7b\" (UniqueName: \"kubernetes.io/projected/fa0b785f-07ed-4382-af03-5709574a77bd-kube-api-access-98q7b\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:02.650064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:02.650033    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"
Oct 30 21:37:02.650064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:02.650052    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs" event={"ID":"fa0b785f-07ed-4382-af03-5709574a77bd","Type":"ContainerDied","Data":"12de3cff82e89e741bfb35da8cab1729683831007d489e8a19e5f32b28fc5b45"}
Oct 30 21:37:02.650328 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:02.650116    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12de3cff82e89e741bfb35da8cab1729683831007d489e8a19e5f32b28fc5b45"
Oct 30 21:37:04.028356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:04.028325    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"]
Oct 30 21:37:04.032930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:04.032907    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af51fca-4nxnz"]
Oct 30 21:37:05.081004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.080973    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a7bb69c-cafa-4838-b17d-39aec6e27a0e" path="/var/lib/kubelet/pods/3a7bb69c-cafa-4838-b17d-39aec6e27a0e/volumes"
Oct 30 21:37:05.210610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.210578    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"]
Oct 30 21:37:05.211643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.211617    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fa0b785f-07ed-4382-af03-5709574a77bd" containerName="revoke-keys"
Oct 30 21:37:05.211643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.211642    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0b785f-07ed-4382-af03-5709574a77bd" containerName="revoke-keys"
Oct 30 21:37:05.211825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.211813    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0b785f-07ed-4382-af03-5709574a77bd" containerName="revoke-keys"
Oct 30 21:37:05.216961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.216939    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.235511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.235461    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"]
Oct 30 21:37:05.243581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.243557    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"]
Oct 30 21:37:05.249375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.249347    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.282218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.282184    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"]
Oct 30 21:37:05.331114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331028    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qz4\" (UniqueName: \"kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-kube-api-access-j9qz4\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.331114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331078    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.331297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331160    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.331297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331186    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.331424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331334    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85acd5b8-9974-4948-9713-ea28737f0d8c-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.331483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331442    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.331524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331506    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.331563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331551    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.331619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331584    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.331703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331683    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.331785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331773    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.331837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331793    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58qg\" (UniqueName: \"kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-kube-api-access-d58qg\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.331837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331833    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.331931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331895    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.331979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.331962    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.332052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.332037    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.332103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.332087    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.332166 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.332125    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.437462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.433791    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.454637    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.454717    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.454779    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.454925    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.455053    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.455082    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58qg\" (UniqueName: \"kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-kube-api-access-d58qg\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.455172    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.455233    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.455424    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.455505    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.455545    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.455609    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.455850    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qz4\" (UniqueName: \"kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-kube-api-access-j9qz4\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.455911    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.455962    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.456004    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.456138    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85acd5b8-9974-4948-9713-ea28737f0d8c-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.469414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.462836    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.469414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.463410    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.469414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.463659    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.469414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.463866    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.469414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.464446    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.490411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.486291    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-data\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.490411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.486505    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-certs\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.490411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.486701    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-credential-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.490411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.486852    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-socket\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.490411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.487265    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.490411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.487853    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85acd5b8-9974-4948-9713-ea28737f0d8c-istiod-ca-cert\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.498110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.498013    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58qg\" (UniqueName: \"kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-kube-api-access-d58qg\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.498322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.498159    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-token\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.499271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.498930    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.499271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.499169    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-envoy\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.499271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.499230    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.500040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.500002    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-podinfo\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.502311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.502222    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qz4\" (UniqueName: \"kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-kube-api-access-j9qz4\") pod \"e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") " pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.551265 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.551230    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:05.574499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.574469    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:05.849306 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:37:05.849260    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85acd5b8_9974_4948_9713_ea28737f0d8c.slice/crio-ebf2452d2f3c1fd67e7137ee21aa0d4f98e2373ebc91d1cc5eadf9c5a71fac79 WatchSource:0}: Error finding container ebf2452d2f3c1fd67e7137ee21aa0d4f98e2373ebc91d1cc5eadf9c5a71fac79: Status 404 returned error can't find the container with id ebf2452d2f3c1fd67e7137ee21aa0d4f98e2373ebc91d1cc5eadf9c5a71fac79
Oct 30 21:37:05.851674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.851650    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"]
Oct 30 21:37:05.853286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.853245    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:37:05.853411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.853333    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:37:05.853411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.853373    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:37:05.888704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.888677    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"]
Oct 30 21:37:05.893033 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:37:05.893005    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e8c87e_a8e0_4051_9459_818dbf9e00e4.slice/crio-9befa53c4ccfac405645f98dfface4c0dd28a10f2083e243dcd4b9b93555fc4f WatchSource:0}: Error finding container 9befa53c4ccfac405645f98dfface4c0dd28a10f2083e243dcd4b9b93555fc4f: Status 404 returned error can't find the container with id 9befa53c4ccfac405645f98dfface4c0dd28a10f2083e243dcd4b9b93555fc4f
Oct 30 21:37:05.896140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.896105    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:37:05.896244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.896168    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:37:05.896244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:05.896197    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:37:06.705506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:06.705474    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2" event={"ID":"80e8c87e-a8e0-4051-9459-818dbf9e00e4","Type":"ContainerStarted","Data":"55a4da94b2829be0cb124af69f5815799b11bdd23170a893dc1e63b108798155"}
Oct 30 21:37:06.706053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:06.705533    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2" event={"ID":"80e8c87e-a8e0-4051-9459-818dbf9e00e4","Type":"ContainerStarted","Data":"9befa53c4ccfac405645f98dfface4c0dd28a10f2083e243dcd4b9b93555fc4f"}
Oct 30 21:37:06.707464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:06.707432    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x" event={"ID":"85acd5b8-9974-4948-9713-ea28737f0d8c","Type":"ContainerStarted","Data":"4231d826ce284be8b4e1eebb48cffcaaa6714b37996e07dd0343f2725c8b6fc4"}
Oct 30 21:37:06.707578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:06.707482    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x" event={"ID":"85acd5b8-9974-4948-9713-ea28737f0d8c","Type":"ContainerStarted","Data":"ebf2452d2f3c1fd67e7137ee21aa0d4f98e2373ebc91d1cc5eadf9c5a71fac79"}
Oct 30 21:37:06.744523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:06.744471    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2" podStartSLOduration=1.744457307 podStartE2EDuration="1.744457307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:06.740269998 +0000 UTC m=+2139.786829165" watchObservedRunningTime="2026-10-30 21:37:06.744457307 +0000 UTC m=+2139.791016452"
Oct 30 21:37:06.772384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:06.772341    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x" podStartSLOduration=1.772329333 podStartE2EDuration="1.772329333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:06.76795278 +0000 UTC m=+2139.814511925" watchObservedRunningTime="2026-10-30 21:37:06.772329333 +0000 UTC m=+2139.818888477"
Oct 30 21:37:07.552581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:07.552549    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:07.553730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:07.553475    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body=
Oct 30 21:37:07.553730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:07.553545    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x" podUID="85acd5b8-9974-4948-9713-ea28737f0d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused"
Oct 30 21:37:07.575193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:07.575165    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:07.576402 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:07.576364    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:37:07.576499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:07.576452    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2" podUID="80e8c87e-a8e0-4051-9459-818dbf9e00e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 21:37:08.552613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:08.552572    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body=
Oct 30 21:37:08.552987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:08.552637    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x" podUID="85acd5b8-9974-4948-9713-ea28737f0d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused"
Oct 30 21:37:08.575384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:08.575350    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:37:08.575547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:08.575424    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2" podUID="80e8c87e-a8e0-4051-9459-818dbf9e00e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 21:37:09.559412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:09.558046    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body=
Oct 30 21:37:09.559412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:09.558117    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x" podUID="85acd5b8-9974-4948-9713-ea28737f0d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused"
Oct 30 21:37:09.582416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:09.580220    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:37:09.582416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:09.580278    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2" podUID="80e8c87e-a8e0-4051-9459-818dbf9e00e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 21:37:09.605418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:09.602868    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"]
Oct 30 21:37:09.641451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:09.640099    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:37:10.300424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.297806    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"]
Oct 30 21:37:10.450419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.450067    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"]
Oct 30 21:37:10.450419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.450184    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:10.454470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.454443    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:37:10.454855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.454498    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:37:10.456566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.456544    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:37:10.456668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.456640    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:37:10.552280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.552197    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body=
Oct 30 21:37:10.552280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.552258    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x" podUID="85acd5b8-9974-4948-9713-ea28737f0d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused"
Oct 30 21:37:10.575678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.575652    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:37:10.576025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.575702    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2" podUID="80e8c87e-a8e0-4051-9459-818dbf9e00e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 21:37:10.577069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.577048    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-080a7479-9bhvs\" (UID: \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:10.577116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.577089    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r427h\" (UniqueName: \"kubernetes.io/projected/e7f17193-1e5c-42b9-b2d9-d959c80b2866-kube-api-access-r427h\") pod \"maas-api-revoke-sub-080a7479-9bhvs\" (UID: \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:10.577155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.577136    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-080a7479-9bhvs\" (UID: \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:10.678710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.678679    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-080a7479-9bhvs\" (UID: \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:10.678710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.678720    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r427h\" (UniqueName: \"kubernetes.io/projected/e7f17193-1e5c-42b9-b2d9-d959c80b2866-kube-api-access-r427h\") pod \"maas-api-revoke-sub-080a7479-9bhvs\" (UID: \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:10.679044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.678763    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-080a7479-9bhvs\" (UID: \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:10.679637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.679609    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-080a7479-9bhvs\" (UID: \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:10.681603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.681583    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-080a7479-9bhvs\" (UID: \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:10.739419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.720342    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r427h\" (UniqueName: \"kubernetes.io/projected/e7f17193-1e5c-42b9-b2d9-d959c80b2866-kube-api-access-r427h\") pod \"maas-api-revoke-sub-080a7479-9bhvs\" (UID: \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:10.770025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:10.769991    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:11.316427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:11.309799    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"]
Oct 30 21:37:11.348340 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:37:11.348290    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f17193_1e5c_42b9_b2d9_d959c80b2866.slice/crio-f707bdff689cc2f11dae43ab1cc773c13a8eb056dcdc710b4b7f5abac3dbf2db WatchSource:0}: Error finding container f707bdff689cc2f11dae43ab1cc773c13a8eb056dcdc710b4b7f5abac3dbf2db: Status 404 returned error can't find the container with id f707bdff689cc2f11dae43ab1cc773c13a8eb056dcdc710b4b7f5abac3dbf2db
Oct 30 21:37:11.552824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:11.552775    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body=
Oct 30 21:37:11.553007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:11.552839    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x" podUID="85acd5b8-9974-4948-9713-ea28737f0d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused"
Oct 30 21:37:11.575423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:11.575339    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:37:11.575423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:11.575384    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2" podUID="80e8c87e-a8e0-4051-9459-818dbf9e00e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 21:37:11.874824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:11.874743    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs" event={"ID":"e7f17193-1e5c-42b9-b2d9-d959c80b2866","Type":"ContainerStarted","Data":"f707bdff689cc2f11dae43ab1cc773c13a8eb056dcdc710b4b7f5abac3dbf2db"}
Oct 30 21:37:12.048535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:12.048052    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"]
Oct 30 21:37:12.056141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:12.056113    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6215f003-lgbzh"]
Oct 30 21:37:12.551984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:12.551952    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused" start-of-body=
Oct 30 21:37:12.552170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:12.552005    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x" podUID="85acd5b8-9974-4948-9713-ea28737f0d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.59:15021/healthz/ready\": dial tcp 10.132.1.59:15021: connect: connection refused"
Oct 30 21:37:12.575120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:12.575090    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused" start-of-body=
Oct 30 21:37:12.575282 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:12.575146    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2" podUID="80e8c87e-a8e0-4051-9459-818dbf9e00e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.60:15021/healthz/ready\": dial tcp 10.132.1.60:15021: connect: connection refused"
Oct 30 21:37:12.918413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:12.917243    4918 generic.go:356] "Generic (PLEG): container finished" podID="e7f17193-1e5c-42b9-b2d9-d959c80b2866" containerID="32c9989db5ba1aac054172177985aa4035913740da565a84a750533bc21a2cbd" exitCode=0
Oct 30 21:37:12.918413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:12.917660    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs" event={"ID":"e7f17193-1e5c-42b9-b2d9-d959c80b2866","Type":"ContainerDied","Data":"32c9989db5ba1aac054172177985aa4035913740da565a84a750533bc21a2cbd"}
Oct 30 21:37:13.084075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:13.084001    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1d645ed-64b5-4c16-9338-b2eacc76be08" path="/var/lib/kubelet/pods/b1d645ed-64b5-4c16-9338-b2eacc76be08/volumes"
Oct 30 21:37:13.591915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:13.591879    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:13.592468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:13.592428    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:13.593707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:13.593686    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:13.593996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:13.593977    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:13.594977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:13.594960    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:37:13.595105 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:13.595045    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:37:14.220530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.220506    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:14.380623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.380592    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-xks-ca\") pod \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\" (UID: \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\") "
Oct 30 21:37:14.380783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.380728    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-openshift-ca\") pod \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\" (UID: \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\") "
Oct 30 21:37:14.380783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.380778    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f17193-1e5c-42b9-b2d9-d959c80b2866-kube-api-access-r427h\" (UniqueName: \"kubernetes.io/projected/e7f17193-1e5c-42b9-b2d9-d959c80b2866-kube-api-access-r427h\") pod \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\" (UID: \"e7f17193-1e5c-42b9-b2d9-d959c80b2866\") "
Oct 30 21:37:14.381142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.381109    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-openshift-ca" pod "e7f17193-1e5c-42b9-b2d9-d959c80b2866" (UID: "e7f17193-1e5c-42b9-b2d9-d959c80b2866"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:14.381525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.381506    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:14.382944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.382916    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-xks-ca" pod "e7f17193-1e5c-42b9-b2d9-d959c80b2866" (UID: "e7f17193-1e5c-42b9-b2d9-d959c80b2866"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:14.383045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.383016    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f17193-1e5c-42b9-b2d9-d959c80b2866-kube-api-access-r427h" pod "e7f17193-1e5c-42b9-b2d9-d959c80b2866" (UID: "e7f17193-1e5c-42b9-b2d9-d959c80b2866"). InnerVolumeSpecName "kube-api-access-r427h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:14.483560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.483525    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r427h\" (UniqueName: \"kubernetes.io/projected/e7f17193-1e5c-42b9-b2d9-d959c80b2866-kube-api-access-r427h\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:14.483560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.483555    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7f17193-1e5c-42b9-b2d9-d959c80b2866-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:14.603944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.603907    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5" podUID="89a3ed3c-4600-4ae7-95a1-cc183597cdd7" containerName="payload-processing" containerID="cri-o://a4fe49267f08f9ce34a20b989be6bf90e7d5ad4eb46ef709b12541ed4971536a" gracePeriod=25
Oct 30 21:37:14.940798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.940767    4918 generic.go:356] "Generic (PLEG): container finished" podID="89a3ed3c-4600-4ae7-95a1-cc183597cdd7" containerID="a4fe49267f08f9ce34a20b989be6bf90e7d5ad4eb46ef709b12541ed4971536a" exitCode=0
Oct 30 21:37:14.940993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.940844    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5" event={"ID":"89a3ed3c-4600-4ae7-95a1-cc183597cdd7","Type":"ContainerDied","Data":"a4fe49267f08f9ce34a20b989be6bf90e7d5ad4eb46ef709b12541ed4971536a"}
Oct 30 21:37:14.942825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.942802    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"
Oct 30 21:37:14.942825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.942818    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs" event={"ID":"e7f17193-1e5c-42b9-b2d9-d959c80b2866","Type":"ContainerDied","Data":"f707bdff689cc2f11dae43ab1cc773c13a8eb056dcdc710b4b7f5abac3dbf2db"}
Oct 30 21:37:14.943008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.942866    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f707bdff689cc2f11dae43ab1cc773c13a8eb056dcdc710b4b7f5abac3dbf2db"
Oct 30 21:37:14.968348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:14.968324    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:37:15.120752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.120668    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-plugins-config-volume\") pod \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") "
Oct 30 21:37:15.120752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.120704    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-kube-api-access-b4jtk\" (UniqueName: \"kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-kube-api-access-b4jtk\") pod \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") "
Oct 30 21:37:15.121002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.120757    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-routing-overlay-volume\") pod \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") "
Oct 30 21:37:15.121002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.120817    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-provider-credentials\") pod \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\" (UID: \"89a3ed3c-4600-4ae7-95a1-cc183597cdd7\") "
Oct 30 21:37:15.121183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.121157    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-plugins-config-volume" pod "89a3ed3c-4600-4ae7-95a1-cc183597cdd7" (UID: "89a3ed3c-4600-4ae7-95a1-cc183597cdd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:15.121183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.121164    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-routing-overlay-volume" pod "89a3ed3c-4600-4ae7-95a1-cc183597cdd7" (UID: "89a3ed3c-4600-4ae7-95a1-cc183597cdd7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:15.121865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.121845    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:15.121865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.121868    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:15.123161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.123141    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-provider-credentials" pod "89a3ed3c-4600-4ae7-95a1-cc183597cdd7" (UID: "89a3ed3c-4600-4ae7-95a1-cc183597cdd7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:15.123244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.123227    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-kube-api-access-b4jtk" pod "89a3ed3c-4600-4ae7-95a1-cc183597cdd7" (UID: "89a3ed3c-4600-4ae7-95a1-cc183597cdd7"). InnerVolumeSpecName "kube-api-access-b4jtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:15.224035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.224011    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:15.224035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.224035    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4jtk\" (UniqueName: \"kubernetes.io/projected/89a3ed3c-4600-4ae7-95a1-cc183597cdd7-kube-api-access-b4jtk\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:15.953939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.953908    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"
Oct 30 21:37:15.954148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.953944    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5" event={"ID":"89a3ed3c-4600-4ae7-95a1-cc183597cdd7","Type":"ContainerDied","Data":"79f0fe3849c99ec78470106fc86ab04aa722d576f4473c2624f1dd2952e447d2"}
Oct 30 21:37:15.954148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.954011    4918 scope.go:122] "RemoveContainer" containerID="a4fe49267f08f9ce34a20b989be6bf90e7d5ad4eb46ef709b12541ed4971536a"
Oct 30 21:37:15.982512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.982483    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"]
Oct 30 21:37:15.987205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:15.987182    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-67ffj59z5"]
Oct 30 21:37:17.085333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:17.085301    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a3ed3c-4600-4ae7-95a1-cc183597cdd7" path="/var/lib/kubelet/pods/89a3ed3c-4600-4ae7-95a1-cc183597cdd7/volumes"
Oct 30 21:37:21.304726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.304690    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"]
Oct 30 21:37:21.305632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.305615    4918 cpu_manager.go:414] "containerMap: removing container" podUID="89a3ed3c-4600-4ae7-95a1-cc183597cdd7" containerName="payload-processing"
Oct 30 21:37:21.305694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.305632    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a3ed3c-4600-4ae7-95a1-cc183597cdd7" containerName="payload-processing"
Oct 30 21:37:21.305694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.305642    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e7f17193-1e5c-42b9-b2d9-d959c80b2866" containerName="revoke-keys"
Oct 30 21:37:21.305694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.305647    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f17193-1e5c-42b9-b2d9-d959c80b2866" containerName="revoke-keys"
Oct 30 21:37:21.305822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.305789    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a3ed3c-4600-4ae7-95a1-cc183597cdd7" containerName="payload-processing"
Oct 30 21:37:21.305822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.305797    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f17193-1e5c-42b9-b2d9-d959c80b2866" containerName="revoke-keys"
Oct 30 21:37:21.309682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.309665    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:21.314189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.314169    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:37:21.315314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.315293    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:37:21.315480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.315311    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:37:21.315548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.315522    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:37:21.318043 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.318015    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"]
Oct 30 21:37:21.428416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.418108    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9278f1d5-smsbm\" (UID: \"2794ef91-9efc-48fd-9baa-8aca4355fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:21.428416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.418460    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9278f1d5-smsbm\" (UID: \"2794ef91-9efc-48fd-9baa-8aca4355fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:21.428416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.418660    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vbk\" (UniqueName: \"kubernetes.io/projected/2794ef91-9efc-48fd-9baa-8aca4355fe64-kube-api-access-x4vbk\") pod \"maas-api-revoke-sub-9278f1d5-smsbm\" (UID: \"2794ef91-9efc-48fd-9baa-8aca4355fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:21.520267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.520231    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vbk\" (UniqueName: \"kubernetes.io/projected/2794ef91-9efc-48fd-9baa-8aca4355fe64-kube-api-access-x4vbk\") pod \"maas-api-revoke-sub-9278f1d5-smsbm\" (UID: \"2794ef91-9efc-48fd-9baa-8aca4355fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:21.520465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.520378    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9278f1d5-smsbm\" (UID: \"2794ef91-9efc-48fd-9baa-8aca4355fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:21.520699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.520683    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9278f1d5-smsbm\" (UID: \"2794ef91-9efc-48fd-9baa-8aca4355fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:21.521325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.521305    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9278f1d5-smsbm\" (UID: \"2794ef91-9efc-48fd-9baa-8aca4355fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:21.523167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.523138    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9278f1d5-smsbm\" (UID: \"2794ef91-9efc-48fd-9baa-8aca4355fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:21.530810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.530781    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vbk\" (UniqueName: \"kubernetes.io/projected/2794ef91-9efc-48fd-9baa-8aca4355fe64-kube-api-access-x4vbk\") pod \"maas-api-revoke-sub-9278f1d5-smsbm\" (UID: \"2794ef91-9efc-48fd-9baa-8aca4355fe64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:21.633173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:21.633100    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:22.134437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:22.106136    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"]
Oct 30 21:37:22.314469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:22.299986    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm" event={"ID":"2794ef91-9efc-48fd-9baa-8aca4355fe64","Type":"ContainerStarted","Data":"bbafab98cd9fc7e8da489c1675a60f5096312f0f5879064318dcff8f6ddcd709"}
Oct 30 21:37:22.359538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:22.349788    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"]
Oct 30 21:37:22.359538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:22.357537    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e4060f4-hd44l"]
Oct 30 21:37:23.191416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:23.189532    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebbec47-f69f-4f8a-997f-4b1be1120e4c" path="/var/lib/kubelet/pods/0ebbec47-f69f-4f8a-997f-4b1be1120e4c/volumes"
Oct 30 21:37:23.360545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:23.357012    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm" event={"ID":"2794ef91-9efc-48fd-9baa-8aca4355fe64","Type":"ContainerStarted","Data":"5acd709be3abcd780e214054ce7a1408a548a8a3bdef0bad6505b356941ed812"}
Oct 30 21:37:23.381477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:23.381425    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm" podStartSLOduration=2.381410696 podStartE2EDuration="2.381410696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:23.377192314 +0000 UTC m=+2156.423751492" watchObservedRunningTime="2026-10-30 21:37:23.381410696 +0000 UTC m=+2156.427969835"
Oct 30 21:37:24.464416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:24.458539    4918 generic.go:356] "Generic (PLEG): container finished" podID="2794ef91-9efc-48fd-9baa-8aca4355fe64" containerID="5acd709be3abcd780e214054ce7a1408a548a8a3bdef0bad6505b356941ed812" exitCode=0
Oct 30 21:37:24.464416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:24.458762    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm" event={"ID":"2794ef91-9efc-48fd-9baa-8aca4355fe64","Type":"ContainerDied","Data":"5acd709be3abcd780e214054ce7a1408a548a8a3bdef0bad6505b356941ed812"}
Oct 30 21:37:25.787339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.787297    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"]
Oct 30 21:37:25.796352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.796322    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:25.798992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.796614    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"]
Oct 30 21:37:25.798992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.796641    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:25.807251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.807200    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"routing-overlay\""
Oct 30 21:37:25.807617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.807598    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"agc-cred-bfb81448bdc2\""
Oct 30 21:37:25.808279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.808248    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"payload-processing-external-model-e2e--d1dc7551-dockercfg-h5b5m\""
Oct 30 21:37:25.808507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.808491    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"openshift-service-ca.crt\""
Oct 30 21:37:25.808660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.808648    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"e2e-cfl-c82625-provider-cred\""
Oct 30 21:37:25.808869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.808854    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a378a2\""
Oct 30 21:37:25.809072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.809036    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"kube-root-ca.crt\""
Oct 30 21:37:25.960756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.960724    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-xks-ca\") pod \"2794ef91-9efc-48fd-9baa-8aca4355fe64\" (UID: \"2794ef91-9efc-48fd-9baa-8aca4355fe64\") "
Oct 30 21:37:25.960992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.960769    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-openshift-ca\") pod \"2794ef91-9efc-48fd-9baa-8aca4355fe64\" (UID: \"2794ef91-9efc-48fd-9baa-8aca4355fe64\") "
Oct 30 21:37:25.961190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.961165    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-openshift-ca" pod "2794ef91-9efc-48fd-9baa-8aca4355fe64" (UID: "2794ef91-9efc-48fd-9baa-8aca4355fe64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:25.961278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.961252    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2794ef91-9efc-48fd-9baa-8aca4355fe64-kube-api-access-x4vbk\" (UniqueName: \"kubernetes.io/projected/2794ef91-9efc-48fd-9baa-8aca4355fe64-kube-api-access-x4vbk\") pod \"2794ef91-9efc-48fd-9baa-8aca4355fe64\" (UID: \"2794ef91-9efc-48fd-9baa-8aca4355fe64\") "
Oct 30 21:37:25.961599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.961575    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6mz6\" (UniqueName: \"kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-kube-api-access-l6mz6\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:25.961770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.961752    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:25.961827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.961785    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:25.961951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.961922    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:25.962314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.962293    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:25.963352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.963329    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-xks-ca" pod "2794ef91-9efc-48fd-9baa-8aca4355fe64" (UID: "2794ef91-9efc-48fd-9baa-8aca4355fe64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:25.963690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:25.963661    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2794ef91-9efc-48fd-9baa-8aca4355fe64-kube-api-access-x4vbk" pod "2794ef91-9efc-48fd-9baa-8aca4355fe64" (UID: "2794ef91-9efc-48fd-9baa-8aca4355fe64"). InnerVolumeSpecName "kube-api-access-x4vbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:26.064086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.064012    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6mz6\" (UniqueName: \"kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-kube-api-access-l6mz6\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:26.064255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.064139    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:26.064255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.064158    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:26.064255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.064207    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:26.064522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.064501    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4vbk\" (UniqueName: \"kubernetes.io/projected/2794ef91-9efc-48fd-9baa-8aca4355fe64-kube-api-access-x4vbk\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:26.064583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.064530    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2794ef91-9efc-48fd-9baa-8aca4355fe64-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:26.064983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.064960    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:26.065027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.064970    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:26.067041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.067017    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:26.074632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.074608    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6mz6\" (UniqueName: \"kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-kube-api-access-l6mz6\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:26.127530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.127500    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:26.422629 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:37:26.422597    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca23f53e_357d_4de2_a35a_c7e882beef6b.slice/crio-35b4bd96ce5002433b15e9068aac8961a7f8f42a06a3365fc08149d582c85bed WatchSource:0}: Error finding container 35b4bd96ce5002433b15e9068aac8961a7f8f42a06a3365fc08149d582c85bed: Status 404 returned error can't find the container with id 35b4bd96ce5002433b15e9068aac8961a7f8f42a06a3365fc08149d582c85bed
Oct 30 21:37:26.422906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.422883    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"]
Oct 30 21:37:26.478805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.478504    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"
Oct 30 21:37:26.478805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.478629    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm" event={"ID":"2794ef91-9efc-48fd-9baa-8aca4355fe64","Type":"ContainerDied","Data":"bbafab98cd9fc7e8da489c1675a60f5096312f0f5879064318dcff8f6ddcd709"}
Oct 30 21:37:26.478805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.478678    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbafab98cd9fc7e8da489c1675a60f5096312f0f5879064318dcff8f6ddcd709"
Oct 30 21:37:26.480703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:26.480672    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw" event={"ID":"ca23f53e-357d-4de2-a35a-c7e882beef6b","Type":"ContainerStarted","Data":"35b4bd96ce5002433b15e9068aac8961a7f8f42a06a3365fc08149d582c85bed"}
Oct 30 21:37:27.492071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:27.492037    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw" event={"ID":"ca23f53e-357d-4de2-a35a-c7e882beef6b","Type":"ContainerStarted","Data":"2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a"}
Oct 30 21:37:27.492524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:27.492131    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:27.514214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:27.514167    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw" podStartSLOduration=2.51415333 podStartE2EDuration="2.51415333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:27.51101293 +0000 UTC m=+2160.557572088" watchObservedRunningTime="2026-10-30 21:37:27.51415333 +0000 UTC m=+2160.560712475"
Oct 30 21:37:29.164500 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:37:29.164466    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:37:31.862264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:31.860633    4918 scope.go:122] "RemoveContainer" containerID="4b6eb0018c3695b426b6025e0a9ce57bc9ca688b66be804a1d638e617248f75e"
Oct 30 21:37:31.880338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:31.880318    4918 scope.go:122] "RemoveContainer" containerID="3fe84d63fe3208b53c91650ce5628e2055321ad460ddcbe13eb614365fefa840"
Oct 30 21:37:31.892465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:31.892444    4918 scope.go:122] "RemoveContainer" containerID="bed3d8852d52e02b6ab3b5724e4e335bdfa07510b6dcb3d3b87b6d3d57b99cec"
Oct 30 21:37:31.907915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:31.907891    4918 scope.go:122] "RemoveContainer" containerID="9f5c5e7bc6482fff55784606cf9fad8ddc9c94389a4965604bdabfe4d58bea25"
Oct 30 21:37:32.339040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:32.339009    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"]
Oct 30 21:37:32.339262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:32.339227    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5" podUID="28bed962-7c0d-4881-b889-bd4d5402a6de" containerName="istio-proxy" containerID="cri-o://97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17" gracePeriod=30
Oct 30 21:37:36.974133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:36.974098    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:37:36.974648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:36.974158    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5" podUID="28bed962-7c0d-4881-b889-bd4d5402a6de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:37:37.410455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.410419    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"]
Oct 30 21:37:37.410639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.410624    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" podUID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" containerName="istio-proxy" containerID="cri-o://215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f" gracePeriod=30
Oct 30 21:37:37.670194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.670161    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:37:37.694726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.694699    4918 generic.go:356] "Generic (PLEG): container finished" podID="28bed962-7c0d-4881-b889-bd4d5402a6de" containerID="97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17" exitCode=0
Oct 30 21:37:37.694898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.694805    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"
Oct 30 21:37:37.694898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.694832    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5" event={"ID":"28bed962-7c0d-4881-b889-bd4d5402a6de","Type":"ContainerDied","Data":"97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17"}
Oct 30 21:37:37.694898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.694891    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5" event={"ID":"28bed962-7c0d-4881-b889-bd4d5402a6de","Type":"ContainerDied","Data":"f773f9dcdedd4193a3cf7911a24a8b25a0a5892fbcd23d84d337455467823f8d"}
Oct 30 21:37:37.695013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.694918    4918 scope.go:122] "RemoveContainer" containerID="97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17"
Oct 30 21:37:37.708491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.708468    4918 scope.go:122] "RemoveContainer" containerID="97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17"
Oct 30 21:37:37.708823 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:37:37.708797    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17\": container with ID starting with 97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17 not found: ID does not exist" containerID="97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17"
Oct 30 21:37:37.708924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.708829    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17"} err="failed to get container status \"97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17\": rpc error: code = NotFound desc = could not find container \"97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17\": container with ID starting with 97ce6f394f9f35e65faa9ca56f4a901de5d5513faad4d107d79cf698c6669a17 not found: ID does not exist"
Oct 30 21:37:37.777994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.777959    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28bed962-7c0d-4881-b889-bd4d5402a6de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bed962-7c0d-4881-b889-bd4d5402a6de-istiod-ca-cert\") pod \"28bed962-7c0d-4881-b889-bd4d5402a6de\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") "
Oct 30 21:37:37.778154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.778143    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-token\" (UniqueName: \"kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-token\") pod \"28bed962-7c0d-4881-b889-bd4d5402a6de\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") "
Oct 30 21:37:37.778208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.778189    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-credential-socket\") pod \"28bed962-7c0d-4881-b889-bd4d5402a6de\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") "
Oct 30 21:37:37.778305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.778289    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-podinfo\") pod \"28bed962-7c0d-4881-b889-bd4d5402a6de\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") "
Oct 30 21:37:37.778353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.778340    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-data\") pod \"28bed962-7c0d-4881-b889-bd4d5402a6de\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") "
Oct 30 21:37:37.778426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.778380    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28bed962-7c0d-4881-b889-bd4d5402a6de-istiod-ca-cert" pod "28bed962-7c0d-4881-b889-bd4d5402a6de" (UID: "28bed962-7c0d-4881-b889-bd4d5402a6de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:37.778485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.778447    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-certs\") pod \"28bed962-7c0d-4881-b889-bd4d5402a6de\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") "
Oct 30 21:37:37.778579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.778556    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-socket\") pod \"28bed962-7c0d-4881-b889-bd4d5402a6de\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") "
Oct 30 21:37:37.778712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.778643    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-envoy\") pod \"28bed962-7c0d-4881-b889-bd4d5402a6de\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") "
Oct 30 21:37:37.778712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.778688    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-credential-socket" pod "28bed962-7c0d-4881-b889-bd4d5402a6de" (UID: "28bed962-7c0d-4881-b889-bd4d5402a6de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:37.778802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.778729    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-kube-api-access-rvslr\" (UniqueName: \"kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-kube-api-access-rvslr\") pod \"28bed962-7c0d-4881-b889-bd4d5402a6de\" (UID: \"28bed962-7c0d-4881-b889-bd4d5402a6de\") "
Oct 30 21:37:37.778897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.778877    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-certs" pod "28bed962-7c0d-4881-b889-bd4d5402a6de" (UID: "28bed962-7c0d-4881-b889-bd4d5402a6de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:37.779109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.779054    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-data" pod "28bed962-7c0d-4881-b889-bd4d5402a6de" (UID: "28bed962-7c0d-4881-b889-bd4d5402a6de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:37.779710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.779616    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-socket" pod "28bed962-7c0d-4881-b889-bd4d5402a6de" (UID: "28bed962-7c0d-4881-b889-bd4d5402a6de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:37.779817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.779746    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:37.779817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.779769    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:37.779817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.779784    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:37.779817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.779798    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28bed962-7c0d-4881-b889-bd4d5402a6de-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:37.781332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.781307    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-token" pod "28bed962-7c0d-4881-b889-bd4d5402a6de" (UID: "28bed962-7c0d-4881-b889-bd4d5402a6de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:37.781595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.781575    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-kube-api-access-rvslr" pod "28bed962-7c0d-4881-b889-bd4d5402a6de" (UID: "28bed962-7c0d-4881-b889-bd4d5402a6de"). InnerVolumeSpecName "kube-api-access-rvslr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:37.781686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.781612    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-podinfo" pod "28bed962-7c0d-4881-b889-bd4d5402a6de" (UID: "28bed962-7c0d-4881-b889-bd4d5402a6de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:37:37.782124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.782093    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-envoy" pod "28bed962-7c0d-4881-b889-bd4d5402a6de" (UID: "28bed962-7c0d-4881-b889-bd4d5402a6de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:37.905427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.904785    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:37.905427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.904819    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:37.905427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.904835    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:37.905427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.904850    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28bed962-7c0d-4881-b889-bd4d5402a6de-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:37.905427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:37.904864    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvslr\" (UniqueName: \"kubernetes.io/projected/28bed962-7c0d-4881-b889-bd4d5402a6de-kube-api-access-rvslr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:38.403663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:38.403459    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"]
Oct 30 21:37:38.453469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:38.444078    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-cn8s5"]
Oct 30 21:37:38.530079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:38.530047    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:37:39.082361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:39.082331    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28bed962-7c0d-4881-b889-bd4d5402a6de" path="/var/lib/kubelet/pods/28bed962-7c0d-4881-b889-bd4d5402a6de/volumes"
Oct 30 21:37:40.048580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.048547    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"]
Oct 30 21:37:40.051164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.051121    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2794ef91-9efc-48fd-9baa-8aca4355fe64" containerName="revoke-keys"
Oct 30 21:37:40.051322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.051308    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2794ef91-9efc-48fd-9baa-8aca4355fe64" containerName="revoke-keys"
Oct 30 21:37:40.051538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.051524    4918 cpu_manager.go:414] "containerMap: removing container" podUID="28bed962-7c0d-4881-b889-bd4d5402a6de" containerName="istio-proxy"
Oct 30 21:37:40.051657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.051645    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28bed962-7c0d-4881-b889-bd4d5402a6de" containerName="istio-proxy"
Oct 30 21:37:40.052178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.052158    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="28bed962-7c0d-4881-b889-bd4d5402a6de" containerName="istio-proxy"
Oct 30 21:37:40.052329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.052317    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2794ef91-9efc-48fd-9baa-8aca4355fe64" containerName="revoke-keys"
Oct 30 21:37:40.061249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.061226    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"]
Oct 30 21:37:40.061499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.061484    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.076341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.076313    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"routing-overlay\""
Oct 30 21:37:40.080447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.079458    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"openshift-service-ca.crt\""
Oct 30 21:37:40.081537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.081515    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"kube-root-ca.crt\""
Oct 30 21:37:40.081660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.081561    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"payload-processing-external-model-e2e--650b641a-dockercfg-s6bl5\""
Oct 30 21:37:40.081660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.081624    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f1ed97\""
Oct 30 21:37:40.081791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.081678    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"agc-cred-c07a671227da\""
Oct 30 21:37:40.176585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.176549    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.176838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.176818    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq92w\" (UniqueName: \"kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-kube-api-access-mq92w\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.176887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.176870    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.177061 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.177049    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.278447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.278374    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.278635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.278472    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.278693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.278658    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq92w\" (UniqueName: \"kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-kube-api-access-mq92w\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.278730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.278700    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.279146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.279125    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.279362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.279345    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.281194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.281178    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.291332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.291305    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq92w\" (UniqueName: \"kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-kube-api-access-mq92w\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.385677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.385590    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:40.837084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:40.837054    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"]
Oct 30 21:37:41.494380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:41.470646    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"]
Oct 30 21:37:41.840513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:41.836955    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:37:41.856784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:41.855918    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:37:41.865509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:41.864292    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:37:41.901296 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:41.897495    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:37:41.976417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:41.945090    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:37:41.976923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:41.966206    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:37:42.015419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:41.966265    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:37:42.052242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.050535    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:37:42.052242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:41.973071    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:37:42.080229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.079077    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq" event={"ID":"cc4e1a61-e664-47f4-8b36-307285aba36d","Type":"ContainerStarted","Data":"4ea7438bf4a5b71ec6351a549d8d84d13b3f269b9c6c32db0a4ad1e419369cdf"}
Oct 30 21:37:42.103476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.080631    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:37:42.725991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.725967    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:37:42.832469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.832344    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-data\") pod \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") "
Oct 30 21:37:42.832469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.832417    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-credential-socket\") pod \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") "
Oct 30 21:37:42.832705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.832491    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-envoy\") pod \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") "
Oct 30 21:37:42.832705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.832617    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-data" pod "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" (UID: "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:42.832705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.832679    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-podinfo\") pod \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") "
Oct 30 21:37:42.832938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.832725    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-socket\") pod \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") "
Oct 30 21:37:42.832938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.832684    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-credential-socket" pod "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" (UID: "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:42.832938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.832764    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-kube-api-access-mf2zd\" (UniqueName: \"kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-kube-api-access-mf2zd\") pod \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") "
Oct 30 21:37:42.832938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.832809    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-certs\") pod \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") "
Oct 30 21:37:42.832938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.832875    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-token\" (UniqueName: \"kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-token\") pod \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") "
Oct 30 21:37:42.832938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.832924    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istiod-ca-cert\") pod \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\" (UID: \"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e\") "
Oct 30 21:37:42.833288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.832988    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-socket" pod "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" (UID: "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:42.833288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.833162    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-certs" pod "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" (UID: "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:42.833427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.833363    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istiod-ca-cert" pod "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" (UID: "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:42.833869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.833799    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:42.833869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.833828    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:42.833869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.833840    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:42.833869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.833848    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:42.833869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.833856    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:42.835055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.835030    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-envoy" pod "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" (UID: "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:42.835448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.835428    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-token" pod "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" (UID: "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:42.835644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.835622    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-podinfo" pod "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" (UID: "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:37:42.835725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.835656    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-kube-api-access-mf2zd" pod "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" (UID: "aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e"). InnerVolumeSpecName "kube-api-access-mf2zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:42.935423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.935361    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:42.935423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.935404    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf2zd\" (UniqueName: \"kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-kube-api-access-mf2zd\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:42.935423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.935415    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:42.935423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:42.935424    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:43.088772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:43.088572    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq" event={"ID":"cc4e1a61-e664-47f4-8b36-307285aba36d","Type":"ContainerStarted","Data":"595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2"}
Oct 30 21:37:43.088772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:43.088654    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:43.090544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:43.090509    4918 generic.go:356] "Generic (PLEG): container finished" podID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" containerID="215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f" exitCode=0
Oct 30 21:37:43.090674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:43.090547    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"
Oct 30 21:37:43.090674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:43.090567    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" event={"ID":"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e","Type":"ContainerDied","Data":"215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f"}
Oct 30 21:37:43.090674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:43.090629    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh" event={"ID":"aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e","Type":"ContainerDied","Data":"e010528470ff0eeb4936c0ce039ccd8da4ea2c591c8ae7a9f0f1d56bcf5b07e0"}
Oct 30 21:37:43.090674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:43.090658    4918 scope.go:122] "RemoveContainer" containerID="215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f"
Oct 30 21:37:43.105048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:43.105029    4918 scope.go:122] "RemoveContainer" containerID="215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f"
Oct 30 21:37:43.105347 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:37:43.105326    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f\": container with ID starting with 215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f not found: ID does not exist" containerID="215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f"
Oct 30 21:37:43.105433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:43.105355    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f"} err="failed to get container status \"215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f\": rpc error: code = NotFound desc = could not find container \"215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f\": container with ID starting with 215ce067d81e37670083f5232870d6ab70c064cdc8494c09ba3a2d7c91d3ba7f not found: ID does not exist"
Oct 30 21:37:43.111653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:43.111609    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq" podStartSLOduration=4.111598272 podStartE2EDuration="4.111598272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:43.108968556 +0000 UTC m=+2176.155527724" watchObservedRunningTime="2026-10-30 21:37:43.111598272 +0000 UTC m=+2176.158157456"
Oct 30 21:37:43.138306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:43.138270    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"]
Oct 30 21:37:43.143897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:43.143872    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-zqlfh"]
Oct 30 21:37:45.084272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:45.084241    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" path="/var/lib/kubelet/pods/aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e/volumes"
Oct 30 21:37:46.471487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:46.471429    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc" podUID="87c78554-01ac-48cc-9998-c8471e310d45" containerName="payload-processing" containerID="cri-o://57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73" gracePeriod=25
Oct 30 21:37:46.858356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:46.858331    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:37:46.978755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:46.978726    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-plugins-config-volume\") pod \"87c78554-01ac-48cc-9998-c8471e310d45\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") "
Oct 30 21:37:46.978905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:46.978868    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-kube-api-access-trfdw\" (UniqueName: \"kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-kube-api-access-trfdw\") pod \"87c78554-01ac-48cc-9998-c8471e310d45\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") "
Oct 30 21:37:46.978975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:46.978905    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-provider-credentials\" (UniqueName: \"kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-provider-credentials\") pod \"87c78554-01ac-48cc-9998-c8471e310d45\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") "
Oct 30 21:37:46.979033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:46.978989    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-routing-overlay-volume\") pod \"87c78554-01ac-48cc-9998-c8471e310d45\" (UID: \"87c78554-01ac-48cc-9998-c8471e310d45\") "
Oct 30 21:37:46.979132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:46.979107    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-plugins-config-volume" pod "87c78554-01ac-48cc-9998-c8471e310d45" (UID: "87c78554-01ac-48cc-9998-c8471e310d45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:46.979470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:46.979439    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-routing-overlay-volume" pod "87c78554-01ac-48cc-9998-c8471e310d45" (UID: "87c78554-01ac-48cc-9998-c8471e310d45"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:46.979954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:46.979935    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:46.980053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:46.979955    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c78554-01ac-48cc-9998-c8471e310d45-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:46.981403 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:46.981365    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-kube-api-access-trfdw" pod "87c78554-01ac-48cc-9998-c8471e310d45" (UID: "87c78554-01ac-48cc-9998-c8471e310d45"). InnerVolumeSpecName "kube-api-access-trfdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:46.981508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:46.981454    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-provider-credentials" pod "87c78554-01ac-48cc-9998-c8471e310d45" (UID: "87c78554-01ac-48cc-9998-c8471e310d45"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:47.082294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.082261    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trfdw\" (UniqueName: \"kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-kube-api-access-trfdw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:47.082294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.082288    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87c78554-01ac-48cc-9998-c8471e310d45-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:47.127307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.127275    4918 generic.go:356] "Generic (PLEG): container finished" podID="87c78554-01ac-48cc-9998-c8471e310d45" containerID="57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73" exitCode=0
Oct 30 21:37:47.127490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.127373    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"
Oct 30 21:37:47.127490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.127448    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc" event={"ID":"87c78554-01ac-48cc-9998-c8471e310d45","Type":"ContainerDied","Data":"57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73"}
Oct 30 21:37:47.127590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.127496    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc" event={"ID":"87c78554-01ac-48cc-9998-c8471e310d45","Type":"ContainerDied","Data":"7173b64d1f353ac8f611b423ae0dbc6876e6bc2cc70043c4e4af2f74e0f3a6fb"}
Oct 30 21:37:47.127590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.127515    4918 scope.go:122] "RemoveContainer" containerID="57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73"
Oct 30 21:37:47.139613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.139595    4918 scope.go:122] "RemoveContainer" containerID="57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73"
Oct 30 21:37:47.139953 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:37:47.139929    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73\": container with ID starting with 57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73 not found: ID does not exist" containerID="57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73"
Oct 30 21:37:47.140068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.139964    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73"} err="failed to get container status \"57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73\": rpc error: code = NotFound desc = could not find container \"57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73\": container with ID starting with 57b9fcd204566b63e075aa84542721864d5004f3fe54ec9d9b8168397458bf73 not found: ID does not exist"
Oct 30 21:37:47.156157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.156132    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"]
Oct 30 21:37:47.165080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.165052    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc"]
Oct 30 21:37:47.758143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.758101    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-86ccrpdpc" podUID="87c78554-01ac-48cc-9998-c8471e310d45" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.53:50052\" within 1s: context deadline exceeded"
Oct 30 21:37:47.758629 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:37:47.758170    4918 logging.go:55] [core] [Channel #6214 SubChannel #6215]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.53:50052", ServerName: "10.132.1.53:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.53:50052: operation was canceled"
Oct 30 21:37:47.799354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.799324    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"]
Oct 30 21:37:47.800217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.800202    4918 cpu_manager.go:414] "containerMap: removing container" podUID="87c78554-01ac-48cc-9998-c8471e310d45" containerName="payload-processing"
Oct 30 21:37:47.800287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.800219    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c78554-01ac-48cc-9998-c8471e310d45" containerName="payload-processing"
Oct 30 21:37:47.800287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.800240    4918 cpu_manager.go:414] "containerMap: removing container" podUID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" containerName="istio-proxy"
Oct 30 21:37:47.800287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.800245    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" containerName="istio-proxy"
Oct 30 21:37:47.800425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.800378    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c78554-01ac-48cc-9998-c8471e310d45" containerName="payload-processing"
Oct 30 21:37:47.800425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.800401    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed0cd50-1dc5-434f-aaa6-14cc9e0dff9e" containerName="istio-proxy"
Oct 30 21:37:47.805578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.805561    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:47.816819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.816796    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:37:47.816949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.816827    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:37:47.816949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.816827    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:37:47.816949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.816908    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:37:47.831995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.831963    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"]
Oct 30 21:37:47.905091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.905060    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-531f9d9e-7rzsh\" (UID: \"6fda6676-8f64-426f-b41c-5f3abd8f240a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:47.905284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.905219    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-531f9d9e-7rzsh\" (UID: \"6fda6676-8f64-426f-b41c-5f3abd8f240a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:47.905339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:47.905331    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gpcf\" (UniqueName: \"kubernetes.io/projected/6fda6676-8f64-426f-b41c-5f3abd8f240a-kube-api-access-9gpcf\") pod \"maas-api-revoke-sub-531f9d9e-7rzsh\" (UID: \"6fda6676-8f64-426f-b41c-5f3abd8f240a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:48.006374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:48.006342    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gpcf\" (UniqueName: \"kubernetes.io/projected/6fda6676-8f64-426f-b41c-5f3abd8f240a-kube-api-access-9gpcf\") pod \"maas-api-revoke-sub-531f9d9e-7rzsh\" (UID: \"6fda6676-8f64-426f-b41c-5f3abd8f240a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:48.006562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:48.006524    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-531f9d9e-7rzsh\" (UID: \"6fda6676-8f64-426f-b41c-5f3abd8f240a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:48.006667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:48.006653    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-531f9d9e-7rzsh\" (UID: \"6fda6676-8f64-426f-b41c-5f3abd8f240a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:48.007174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:48.007149    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-531f9d9e-7rzsh\" (UID: \"6fda6676-8f64-426f-b41c-5f3abd8f240a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:48.009294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:48.009241    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-531f9d9e-7rzsh\" (UID: \"6fda6676-8f64-426f-b41c-5f3abd8f240a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:48.014985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:48.014965    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gpcf\" (UniqueName: \"kubernetes.io/projected/6fda6676-8f64-426f-b41c-5f3abd8f240a-kube-api-access-9gpcf\") pod \"maas-api-revoke-sub-531f9d9e-7rzsh\" (UID: \"6fda6676-8f64-426f-b41c-5f3abd8f240a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:48.120262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:48.120220    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:48.419271 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:37:48.419241    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fda6676_8f64_426f_b41c_5f3abd8f240a.slice/crio-96dd4568e67ec0d81bbff3542350730e64fbaebffb7bb36ad312685366514938 WatchSource:0}: Error finding container 96dd4568e67ec0d81bbff3542350730e64fbaebffb7bb36ad312685366514938: Status 404 returned error can't find the container with id 96dd4568e67ec0d81bbff3542350730e64fbaebffb7bb36ad312685366514938
Oct 30 21:37:48.419806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:48.419781    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"]
Oct 30 21:37:49.080630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:49.080553    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c78554-01ac-48cc-9998-c8471e310d45" path="/var/lib/kubelet/pods/87c78554-01ac-48cc-9998-c8471e310d45/volumes"
Oct 30 21:37:49.150110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:49.150034    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh" event={"ID":"6fda6676-8f64-426f-b41c-5f3abd8f240a","Type":"ContainerStarted","Data":"c024b4c647e84c4101be529330a4009c6d6570b23cedf28ef1ce5a37e4468ca6"}
Oct 30 21:37:49.150329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:49.150137    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh" event={"ID":"6fda6676-8f64-426f-b41c-5f3abd8f240a","Type":"ContainerStarted","Data":"96dd4568e67ec0d81bbff3542350730e64fbaebffb7bb36ad312685366514938"}
Oct 30 21:37:49.168879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:49.168844    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"]
Oct 30 21:37:49.169077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:49.169055    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99" podUID="a4849090-13cf-4a35-8160-86860f71d19c" containerName="istio-proxy" containerID="cri-o://11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573" gracePeriod=30
Oct 30 21:37:49.173745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:49.173721    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"]
Oct 30 21:37:49.173913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:49.173895    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy" containerID="cri-o://3a8948db74acbcb88736c86faaebf1a8c5958ac4bf6d78fc3084dadcf63edd56" gracePeriod=30
Oct 30 21:37:49.208351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:49.208301    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh" podStartSLOduration=2.208287891 podStartE2EDuration="2.208287891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:49.203810813 +0000 UTC m=+2182.250369975" watchObservedRunningTime="2026-10-30 21:37:49.208287891 +0000 UTC m=+2182.254847425"
Oct 30 21:37:50.160353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:50.160318    4918 generic.go:356] "Generic (PLEG): container finished" podID="6fda6676-8f64-426f-b41c-5f3abd8f240a" containerID="c024b4c647e84c4101be529330a4009c6d6570b23cedf28ef1ce5a37e4468ca6" exitCode=0
Oct 30 21:37:50.160747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:50.160412    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh" event={"ID":"6fda6676-8f64-426f-b41c-5f3abd8f240a","Type":"ContainerDied","Data":"c024b4c647e84c4101be529330a4009c6d6570b23cedf28ef1ce5a37e4468ca6"}
Oct 30 21:37:51.425463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:51.425440    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:51.524636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:51.524536    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-openshift-ca\") pod \"6fda6676-8f64-426f-b41c-5f3abd8f240a\" (UID: \"6fda6676-8f64-426f-b41c-5f3abd8f240a\") "
Oct 30 21:37:51.524870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:51.524849    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fda6676-8f64-426f-b41c-5f3abd8f240a-kube-api-access-9gpcf\" (UniqueName: \"kubernetes.io/projected/6fda6676-8f64-426f-b41c-5f3abd8f240a-kube-api-access-9gpcf\") pod \"6fda6676-8f64-426f-b41c-5f3abd8f240a\" (UID: \"6fda6676-8f64-426f-b41c-5f3abd8f240a\") "
Oct 30 21:37:51.524984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:51.524946    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-openshift-ca" pod "6fda6676-8f64-426f-b41c-5f3abd8f240a" (UID: "6fda6676-8f64-426f-b41c-5f3abd8f240a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:51.524984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:51.524955    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-xks-ca\") pod \"6fda6676-8f64-426f-b41c-5f3abd8f240a\" (UID: \"6fda6676-8f64-426f-b41c-5f3abd8f240a\") "
Oct 30 21:37:51.525697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:51.525673    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:51.527153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:51.527129    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-xks-ca" pod "6fda6676-8f64-426f-b41c-5f3abd8f240a" (UID: "6fda6676-8f64-426f-b41c-5f3abd8f240a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:37:51.527633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:51.527609    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fda6676-8f64-426f-b41c-5f3abd8f240a-kube-api-access-9gpcf" pod "6fda6676-8f64-426f-b41c-5f3abd8f240a" (UID: "6fda6676-8f64-426f-b41c-5f3abd8f240a"). InnerVolumeSpecName "kube-api-access-9gpcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:51.627916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:51.627883    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gpcf\" (UniqueName: \"kubernetes.io/projected/6fda6676-8f64-426f-b41c-5f3abd8f240a-kube-api-access-9gpcf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:51.627916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:51.627914    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fda6676-8f64-426f-b41c-5f3abd8f240a-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:52.236498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:52.236455    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh" event={"ID":"6fda6676-8f64-426f-b41c-5f3abd8f240a","Type":"ContainerDied","Data":"96dd4568e67ec0d81bbff3542350730e64fbaebffb7bb36ad312685366514938"}
Oct 30 21:37:52.236498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:52.236491    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"
Oct 30 21:37:52.236757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:52.236525    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96dd4568e67ec0d81bbff3542350730e64fbaebffb7bb36ad312685366514938"
Oct 30 21:37:54.104602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.104573    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:37:54.349271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.349242    4918 generic.go:356] "Generic (PLEG): container finished" podID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerID="3a8948db74acbcb88736c86faaebf1a8c5958ac4bf6d78fc3084dadcf63edd56" exitCode=0
Oct 30 21:37:54.349427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.349311    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" event={"ID":"8ee5b7c3-c5f9-4b08-83e5-026d8df893da","Type":"ContainerDied","Data":"3a8948db74acbcb88736c86faaebf1a8c5958ac4bf6d78fc3084dadcf63edd56"}
Oct 30 21:37:54.349504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.349482    4918 scope.go:122] "RemoveContainer" containerID="f435fd66414b4c4f7b2ebc73d6407d380591d41f1bb1eca3fa5bde0287efb7c8"
Oct 30 21:37:54.597307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.597285    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:37:54.600280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.600261    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:37:54.720917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.720890    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4849090-13cf-4a35-8160-86860f71d19c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4849090-13cf-4a35-8160-86860f71d19c-istio-podinfo\") pod \"a4849090-13cf-4a35-8160-86860f71d19c\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") "
Oct 30 21:37:54.721125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.720986    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4849090-13cf-4a35-8160-86860f71d19c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4849090-13cf-4a35-8160-86860f71d19c-istiod-ca-cert\") pod \"a4849090-13cf-4a35-8160-86860f71d19c\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") "
Oct 30 21:37:54.721125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721086    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-credential-socket\") pod \"a4849090-13cf-4a35-8160-86860f71d19c\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") "
Oct 30 21:37:54.721256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721137    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-kube-api-access-74g5j\" (UniqueName: \"kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-kube-api-access-74g5j\") pod \"a4849090-13cf-4a35-8160-86860f71d19c\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") "
Oct 30 21:37:54.721256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721183    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-certs\") pod \"a4849090-13cf-4a35-8160-86860f71d19c\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") "
Oct 30 21:37:54.721256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721227    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-podinfo\") pod \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") "
Oct 30 21:37:54.721510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721268    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-data\") pod \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") "
Oct 30 21:37:54.721510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721297    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-istio-token\" (UniqueName: \"kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-istio-token\") pod \"a4849090-13cf-4a35-8160-86860f71d19c\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") "
Oct 30 21:37:54.721510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721321    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-credential-socket" pod "a4849090-13cf-4a35-8160-86860f71d19c" (UID: "a4849090-13cf-4a35-8160-86860f71d19c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:54.721510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721360    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istiod-ca-cert\") pod \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") "
Oct 30 21:37:54.721510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721362    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4849090-13cf-4a35-8160-86860f71d19c-istiod-ca-cert" pod "a4849090-13cf-4a35-8160-86860f71d19c" (UID: "a4849090-13cf-4a35-8160-86860f71d19c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:54.721510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721418    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-token\" (UniqueName: \"kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-token\") pod \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") "
Oct 30 21:37:54.721510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721470    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-envoy\") pod \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") "
Oct 30 21:37:54.722056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721564    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-data" pod "8ee5b7c3-c5f9-4b08-83e5-026d8df893da" (UID: "8ee5b7c3-c5f9-4b08-83e5-026d8df893da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:54.722056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721575    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-certs\") pod \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") "
Oct 30 21:37:54.722056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721624    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-socket\") pod \"a4849090-13cf-4a35-8160-86860f71d19c\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") "
Oct 30 21:37:54.722056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721720    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-envoy\") pod \"a4849090-13cf-4a35-8160-86860f71d19c\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") "
Oct 30 21:37:54.722056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721749    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-data\") pod \"a4849090-13cf-4a35-8160-86860f71d19c\" (UID: \"a4849090-13cf-4a35-8160-86860f71d19c\") "
Oct 30 21:37:54.722056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721829    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-credential-socket\") pod \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") "
Oct 30 21:37:54.722056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721859    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-kube-api-access-djvsw\" (UniqueName: \"kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-kube-api-access-djvsw\") pod \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") "
Oct 30 21:37:54.722056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721874    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istiod-ca-cert" pod "8ee5b7c3-c5f9-4b08-83e5-026d8df893da" (UID: "8ee5b7c3-c5f9-4b08-83e5-026d8df893da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:37:54.722056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.721916    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-socket\") pod \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\" (UID: \"8ee5b7c3-c5f9-4b08-83e5-026d8df893da\") "
Oct 30 21:37:54.722728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.722702    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-certs" pod "a4849090-13cf-4a35-8160-86860f71d19c" (UID: "a4849090-13cf-4a35-8160-86860f71d19c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:54.723121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.722921    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-socket" pod "8ee5b7c3-c5f9-4b08-83e5-026d8df893da" (UID: "8ee5b7c3-c5f9-4b08-83e5-026d8df893da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:54.723121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.723024    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4849090-13cf-4a35-8160-86860f71d19c-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.723121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.723049    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.723121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.723064    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.723121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.723079    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.723121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.723093    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.724097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.723272    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-socket" pod "a4849090-13cf-4a35-8160-86860f71d19c" (UID: "a4849090-13cf-4a35-8160-86860f71d19c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:54.724312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.724276    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-data" pod "a4849090-13cf-4a35-8160-86860f71d19c" (UID: "a4849090-13cf-4a35-8160-86860f71d19c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:54.724513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.724486    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-certs" pod "8ee5b7c3-c5f9-4b08-83e5-026d8df893da" (UID: "8ee5b7c3-c5f9-4b08-83e5-026d8df893da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:54.726349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.726327    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-kube-api-access-djvsw" pod "8ee5b7c3-c5f9-4b08-83e5-026d8df893da" (UID: "8ee5b7c3-c5f9-4b08-83e5-026d8df893da"). InnerVolumeSpecName "kube-api-access-djvsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:54.726706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.726683    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-envoy" pod "8ee5b7c3-c5f9-4b08-83e5-026d8df893da" (UID: "8ee5b7c3-c5f9-4b08-83e5-026d8df893da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:54.727485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.727460    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-envoy" pod "a4849090-13cf-4a35-8160-86860f71d19c" (UID: "a4849090-13cf-4a35-8160-86860f71d19c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:54.727572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.727509    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-credential-socket" pod "8ee5b7c3-c5f9-4b08-83e5-026d8df893da" (UID: "8ee5b7c3-c5f9-4b08-83e5-026d8df893da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:37:54.727744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.727719    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4849090-13cf-4a35-8160-86860f71d19c-istio-podinfo" pod "a4849090-13cf-4a35-8160-86860f71d19c" (UID: "a4849090-13cf-4a35-8160-86860f71d19c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:37:54.727862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.727761    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-kube-api-access-74g5j" pod "a4849090-13cf-4a35-8160-86860f71d19c" (UID: "a4849090-13cf-4a35-8160-86860f71d19c"). InnerVolumeSpecName "kube-api-access-74g5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:54.729654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.729634    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-istio-token" pod "a4849090-13cf-4a35-8160-86860f71d19c" (UID: "a4849090-13cf-4a35-8160-86860f71d19c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:54.729890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.729872    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-podinfo" pod "8ee5b7c3-c5f9-4b08-83e5-026d8df893da" (UID: "8ee5b7c3-c5f9-4b08-83e5-026d8df893da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:37:54.730754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.730734    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-token" pod "8ee5b7c3-c5f9-4b08-83e5-026d8df893da" (UID: "8ee5b7c3-c5f9-4b08-83e5-026d8df893da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:37:54.824756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824719    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.824756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824751    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4849090-13cf-4a35-8160-86860f71d19c-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.824977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824766    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74g5j\" (UniqueName: \"kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-kube-api-access-74g5j\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.824977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824780    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.824977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824793    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4849090-13cf-4a35-8160-86860f71d19c-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.824977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824803    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.824977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824815    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.824977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824827    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.824977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824840    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.824977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824853    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.824977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824865    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4849090-13cf-4a35-8160-86860f71d19c-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.824977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824878    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:54.824977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:54.824891    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djvsw\" (UniqueName: \"kubernetes.io/projected/8ee5b7c3-c5f9-4b08-83e5-026d8df893da-kube-api-access-djvsw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:37:55.359191 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.359104    4918 generic.go:356] "Generic (PLEG): container finished" podID="a4849090-13cf-4a35-8160-86860f71d19c" containerID="11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573" exitCode=0
Oct 30 21:37:55.359633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.359208    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"
Oct 30 21:37:55.359633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.359226    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99" event={"ID":"a4849090-13cf-4a35-8160-86860f71d19c","Type":"ContainerDied","Data":"11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573"}
Oct 30 21:37:55.359633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.359282    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99" event={"ID":"a4849090-13cf-4a35-8160-86860f71d19c","Type":"ContainerDied","Data":"1b6b33451382a6e7f78496e9b4ab10430a97fc53f799f8a7d84592d5518e331b"}
Oct 30 21:37:55.359633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.359314    4918 scope.go:122] "RemoveContainer" containerID="11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573"
Oct 30 21:37:55.361187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.361152    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"
Oct 30 21:37:55.361187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.361166    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss" event={"ID":"8ee5b7c3-c5f9-4b08-83e5-026d8df893da","Type":"ContainerDied","Data":"6271609ef034046f0242c2d022e55510a82855ce778a9a5deb98b5d234724ad2"}
Oct 30 21:37:55.372672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.372651    4918 scope.go:122] "RemoveContainer" containerID="11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573"
Oct 30 21:37:55.372962 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:37:55.372937    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573\": container with ID starting with 11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573 not found: ID does not exist" containerID="11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573"
Oct 30 21:37:55.373055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.372973    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573"} err="failed to get container status \"11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573\": rpc error: code = NotFound desc = could not find container \"11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573\": container with ID starting with 11b5b006ed14453e139c91e7a0a8cbd26807dfef4a5f0dc3712b068266e9f573 not found: ID does not exist"
Oct 30 21:37:55.373055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.372994    4918 scope.go:122] "RemoveContainer" containerID="3a8948db74acbcb88736c86faaebf1a8c5958ac4bf6d78fc3084dadcf63edd56"
Oct 30 21:37:55.388980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.388958    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"]
Oct 30 21:37:55.396479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.396412    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-qrhss"]
Oct 30 21:37:55.416196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.416170    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"]
Oct 30 21:37:55.423075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:55.423054    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-7hd99"]
Oct 30 21:37:57.080828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:57.080799    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" path="/var/lib/kubelet/pods/8ee5b7c3-c5f9-4b08-83e5-026d8df893da/volumes"
Oct 30 21:37:57.081307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:57.081293    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4849090-13cf-4a35-8160-86860f71d19c" path="/var/lib/kubelet/pods/a4849090-13cf-4a35-8160-86860f71d19c/volumes"
Oct 30 21:37:59.160848 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:37:59.160811    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:37:59.195638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.195610    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"]
Oct 30 21:37:59.196480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.196462    4918 cpu_manager.go:414] "containerMap: removing container" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy"
Oct 30 21:37:59.196571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.196481    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy"
Oct 30 21:37:59.196571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.196501    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a4849090-13cf-4a35-8160-86860f71d19c" containerName="istio-proxy"
Oct 30 21:37:59.196571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.196506    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4849090-13cf-4a35-8160-86860f71d19c" containerName="istio-proxy"
Oct 30 21:37:59.196571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.196538    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6fda6676-8f64-426f-b41c-5f3abd8f240a" containerName="revoke-keys"
Oct 30 21:37:59.196571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.196543    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fda6676-8f64-426f-b41c-5f3abd8f240a" containerName="revoke-keys"
Oct 30 21:37:59.196833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.196697    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy"
Oct 30 21:37:59.196833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.196709    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4849090-13cf-4a35-8160-86860f71d19c" containerName="istio-proxy"
Oct 30 21:37:59.196833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.196719    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fda6676-8f64-426f-b41c-5f3abd8f240a" containerName="revoke-keys"
Oct 30 21:37:59.202559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.202536    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:37:59.212309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.212171    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:37:59.213045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.213020    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:37:59.217081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.217058    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"]
Oct 30 21:37:59.255020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.254992    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:37:59.255243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.255073    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:37:59.319398 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.319365    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937adfda-dghmt\" (UID: \"29b9716a-4f3f-4c1d-8366-58967d60cddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:37:59.319569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.319425    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv587\" (UniqueName: \"kubernetes.io/projected/29b9716a-4f3f-4c1d-8366-58967d60cddd-kube-api-access-xv587\") pod \"maas-api-revoke-sub-937adfda-dghmt\" (UID: \"29b9716a-4f3f-4c1d-8366-58967d60cddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:37:59.319687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.319666    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937adfda-dghmt\" (UID: \"29b9716a-4f3f-4c1d-8366-58967d60cddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:37:59.421673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.421586    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937adfda-dghmt\" (UID: \"29b9716a-4f3f-4c1d-8366-58967d60cddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:37:59.421673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.421646    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv587\" (UniqueName: \"kubernetes.io/projected/29b9716a-4f3f-4c1d-8366-58967d60cddd-kube-api-access-xv587\") pod \"maas-api-revoke-sub-937adfda-dghmt\" (UID: \"29b9716a-4f3f-4c1d-8366-58967d60cddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:37:59.421921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.421873    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937adfda-dghmt\" (UID: \"29b9716a-4f3f-4c1d-8366-58967d60cddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:37:59.422259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.422240    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937adfda-dghmt\" (UID: \"29b9716a-4f3f-4c1d-8366-58967d60cddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:37:59.424405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.424375    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937adfda-dghmt\" (UID: \"29b9716a-4f3f-4c1d-8366-58967d60cddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:37:59.432313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.432288    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv587\" (UniqueName: \"kubernetes.io/projected/29b9716a-4f3f-4c1d-8366-58967d60cddd-kube-api-access-xv587\") pod \"maas-api-revoke-sub-937adfda-dghmt\" (UID: \"29b9716a-4f3f-4c1d-8366-58967d60cddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:37:59.522149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.522115    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:37:59.900746 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:37:59.900713    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b9716a_4f3f_4c1d_8366_58967d60cddd.slice/crio-9064b9e177df619e74421e0cc92e80e4d0a514846f8d4294dc56f2cbfdc65a17 WatchSource:0}: Error finding container 9064b9e177df619e74421e0cc92e80e4d0a514846f8d4294dc56f2cbfdc65a17: Status 404 returned error can't find the container with id 9064b9e177df619e74421e0cc92e80e4d0a514846f8d4294dc56f2cbfdc65a17
Oct 30 21:37:59.902132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:37:59.902106    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"]
Oct 30 21:38:00.408632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:00.408599    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt" event={"ID":"29b9716a-4f3f-4c1d-8366-58967d60cddd","Type":"ContainerStarted","Data":"8a2fd7d5f8a486b745ca3b1b64ba5ac543407c79d64c8470ca9a8564f62b7f4e"}
Oct 30 21:38:00.409048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:00.408661    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt" event={"ID":"29b9716a-4f3f-4c1d-8366-58967d60cddd","Type":"ContainerStarted","Data":"9064b9e177df619e74421e0cc92e80e4d0a514846f8d4294dc56f2cbfdc65a17"}
Oct 30 21:38:00.434170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:00.434112    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt" podStartSLOduration=1.434097779 podStartE2EDuration="1.434097779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:00.42835109 +0000 UTC m=+2193.474910261" watchObservedRunningTime="2026-10-30 21:38:00.434097779 +0000 UTC m=+2193.480656924"
Oct 30 21:38:01.460914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:01.460820    4918 generic.go:356] "Generic (PLEG): container finished" podID="29b9716a-4f3f-4c1d-8366-58967d60cddd" containerID="8a2fd7d5f8a486b745ca3b1b64ba5ac543407c79d64c8470ca9a8564f62b7f4e" exitCode=0
Oct 30 21:38:01.461594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:01.461571    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt" event={"ID":"29b9716a-4f3f-4c1d-8366-58967d60cddd","Type":"ContainerDied","Data":"8a2fd7d5f8a486b745ca3b1b64ba5ac543407c79d64c8470ca9a8564f62b7f4e"}
Oct 30 21:38:02.755940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:02.755870    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:38:02.838561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:02.838532    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-openshift-ca\") pod \"29b9716a-4f3f-4c1d-8366-58967d60cddd\" (UID: \"29b9716a-4f3f-4c1d-8366-58967d60cddd\") "
Oct 30 21:38:02.838752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:02.838620    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b9716a-4f3f-4c1d-8366-58967d60cddd-kube-api-access-xv587\" (UniqueName: \"kubernetes.io/projected/29b9716a-4f3f-4c1d-8366-58967d60cddd-kube-api-access-xv587\") pod \"29b9716a-4f3f-4c1d-8366-58967d60cddd\" (UID: \"29b9716a-4f3f-4c1d-8366-58967d60cddd\") "
Oct 30 21:38:02.838923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:02.838877    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-xks-ca\") pod \"29b9716a-4f3f-4c1d-8366-58967d60cddd\" (UID: \"29b9716a-4f3f-4c1d-8366-58967d60cddd\") "
Oct 30 21:38:02.839034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:02.838874    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-openshift-ca" pod "29b9716a-4f3f-4c1d-8366-58967d60cddd" (UID: "29b9716a-4f3f-4c1d-8366-58967d60cddd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:02.839916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:02.839886    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:02.841085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:02.841057    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b9716a-4f3f-4c1d-8366-58967d60cddd-kube-api-access-xv587" pod "29b9716a-4f3f-4c1d-8366-58967d60cddd" (UID: "29b9716a-4f3f-4c1d-8366-58967d60cddd"). InnerVolumeSpecName "kube-api-access-xv587". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:02.841219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:02.841119    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-xks-ca" pod "29b9716a-4f3f-4c1d-8366-58967d60cddd" (UID: "29b9716a-4f3f-4c1d-8366-58967d60cddd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:02.941623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:02.941589    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b9716a-4f3f-4c1d-8366-58967d60cddd-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:02.941623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:02.941621    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv587\" (UniqueName: \"kubernetes.io/projected/29b9716a-4f3f-4c1d-8366-58967d60cddd-kube-api-access-xv587\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:03.496244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:03.496218    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"
Oct 30 21:38:03.496441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:03.496215    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt" event={"ID":"29b9716a-4f3f-4c1d-8366-58967d60cddd","Type":"ContainerDied","Data":"9064b9e177df619e74421e0cc92e80e4d0a514846f8d4294dc56f2cbfdc65a17"}
Oct 30 21:38:03.496441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:03.496344    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9064b9e177df619e74421e0cc92e80e4d0a514846f8d4294dc56f2cbfdc65a17"
Oct 30 21:38:05.932887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:05.932853    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"]
Oct 30 21:38:05.933756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:05.933740    4918 cpu_manager.go:414] "containerMap: removing container" podUID="29b9716a-4f3f-4c1d-8366-58967d60cddd" containerName="revoke-keys"
Oct 30 21:38:05.933814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:05.933758    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b9716a-4f3f-4c1d-8366-58967d60cddd" containerName="revoke-keys"
Oct 30 21:38:05.933814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:05.933791    4918 cpu_manager.go:414] "containerMap: removing container" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy"
Oct 30 21:38:05.933814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:05.933796    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy"
Oct 30 21:38:05.933937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:05.933927    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee5b7c3-c5f9-4b08-83e5-026d8df893da" containerName="istio-proxy"
Oct 30 21:38:05.933977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:05.933950    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b9716a-4f3f-4c1d-8366-58967d60cddd" containerName="revoke-keys"
Oct 30 21:38:05.937726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:05.937707    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:05.944798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:05.944772    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"agc-cred-920fc7e8729c\""
Oct 30 21:38:05.946314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:05.946287    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"]
Oct 30 21:38:06.051727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.051696    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.051912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.051811    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.051981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.051965    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxcdd\" (UniqueName: \"kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-kube-api-access-bxcdd\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.052067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.052056    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.153230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.153191    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxcdd\" (UniqueName: \"kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-kube-api-access-bxcdd\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.153440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.153314    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.153524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.153504    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.153614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.153601    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.154201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.154172    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.154316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.154220    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.156109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.156090    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.161637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.161612    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxcdd\" (UniqueName: \"kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-kube-api-access-bxcdd\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.260016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.259978    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:06.848762 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:38:06.848730    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b1dc8f_ad62_4490_b18d_0213bd1f137e.slice/crio-2708cebfbc927ab53c2f3b2c37cdd0366b816d33e7d66ee9ca509a4bcf5cd3b1 WatchSource:0}: Error finding container 2708cebfbc927ab53c2f3b2c37cdd0366b816d33e7d66ee9ca509a4bcf5cd3b1: Status 404 returned error can't find the container with id 2708cebfbc927ab53c2f3b2c37cdd0366b816d33e7d66ee9ca509a4bcf5cd3b1
Oct 30 21:38:06.849620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:06.849595    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"]
Oct 30 21:38:07.606862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:07.606820    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm" event={"ID":"25b1dc8f-ad62-4490-b18d-0213bd1f137e","Type":"ContainerStarted","Data":"05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13"}
Oct 30 21:38:07.607317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:07.606885    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm" event={"ID":"25b1dc8f-ad62-4490-b18d-0213bd1f137e","Type":"ContainerStarted","Data":"2708cebfbc927ab53c2f3b2c37cdd0366b816d33e7d66ee9ca509a4bcf5cd3b1"}
Oct 30 21:38:07.607317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:07.606915    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:07.626528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:07.626485    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm" podStartSLOduration=2.626470782 podStartE2EDuration="2.626470782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:07.624991856 +0000 UTC m=+2200.671551021" watchObservedRunningTime="2026-10-30 21:38:07.626470782 +0000 UTC m=+2200.673029963"
Oct 30 21:38:09.686465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.685590    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"]
Oct 30 21:38:09.691039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.691016    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:09.693486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.693458    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"]
Oct 30 21:38:09.695612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.695580    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:38:09.698448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.698378    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:38:09.699127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.699105    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:38:09.699127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.699122    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:38:09.773485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.773456    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d04f9e5-4726-443a-bf43-674986770835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c189cba-g7gj7\" (UID: \"7d04f9e5-4726-443a-bf43-674986770835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:09.773665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.773645    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d04f9e5-4726-443a-bf43-674986770835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c189cba-g7gj7\" (UID: \"7d04f9e5-4726-443a-bf43-674986770835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:09.773844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.773826    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwxxn\" (UniqueName: \"kubernetes.io/projected/7d04f9e5-4726-443a-bf43-674986770835-kube-api-access-jwxxn\") pod \"maas-api-revoke-sub-2c189cba-g7gj7\" (UID: \"7d04f9e5-4726-443a-bf43-674986770835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:09.875672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.875632    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwxxn\" (UniqueName: \"kubernetes.io/projected/7d04f9e5-4726-443a-bf43-674986770835-kube-api-access-jwxxn\") pod \"maas-api-revoke-sub-2c189cba-g7gj7\" (UID: \"7d04f9e5-4726-443a-bf43-674986770835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:09.875848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.875785    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d04f9e5-4726-443a-bf43-674986770835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c189cba-g7gj7\" (UID: \"7d04f9e5-4726-443a-bf43-674986770835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:09.875969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.875952    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d04f9e5-4726-443a-bf43-674986770835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c189cba-g7gj7\" (UID: \"7d04f9e5-4726-443a-bf43-674986770835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:09.876653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.876578    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d04f9e5-4726-443a-bf43-674986770835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c189cba-g7gj7\" (UID: \"7d04f9e5-4726-443a-bf43-674986770835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:09.878527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.878504    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d04f9e5-4726-443a-bf43-674986770835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c189cba-g7gj7\" (UID: \"7d04f9e5-4726-443a-bf43-674986770835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:09.885035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:09.885011    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwxxn\" (UniqueName: \"kubernetes.io/projected/7d04f9e5-4726-443a-bf43-674986770835-kube-api-access-jwxxn\") pod \"maas-api-revoke-sub-2c189cba-g7gj7\" (UID: \"7d04f9e5-4726-443a-bf43-674986770835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:10.008581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:10.008550    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:10.370435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:10.369696    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"]
Oct 30 21:38:10.374880 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:38:10.374841    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d04f9e5_4726_443a_bf43_674986770835.slice/crio-a980824901cf9fb128f4e2ba7c4217e95c9aab1f2b6e277a0f1bfd383158f36c WatchSource:0}: Error finding container a980824901cf9fb128f4e2ba7c4217e95c9aab1f2b6e277a0f1bfd383158f36c: Status 404 returned error can't find the container with id a980824901cf9fb128f4e2ba7c4217e95c9aab1f2b6e277a0f1bfd383158f36c
Oct 30 21:38:10.681055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:10.680965    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7" event={"ID":"7d04f9e5-4726-443a-bf43-674986770835","Type":"ContainerStarted","Data":"a980824901cf9fb128f4e2ba7c4217e95c9aab1f2b6e277a0f1bfd383158f36c"}
Oct 30 21:38:11.694735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:11.694657    4918 generic.go:356] "Generic (PLEG): container finished" podID="7d04f9e5-4726-443a-bf43-674986770835" containerID="9fb516aec342c1b6d7f77a5c291fa299c21ac2820fc10b55b5f0a1d16f1f91b2" exitCode=0
Oct 30 21:38:11.695085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:11.694738    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7" event={"ID":"7d04f9e5-4726-443a-bf43-674986770835","Type":"ContainerDied","Data":"9fb516aec342c1b6d7f77a5c291fa299c21ac2820fc10b55b5f0a1d16f1f91b2"}
Oct 30 21:38:12.902093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:12.902069    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:12.963603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:12.963150    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d04f9e5-4726-443a-bf43-674986770835-kube-api-access-jwxxn\" (UniqueName: \"kubernetes.io/projected/7d04f9e5-4726-443a-bf43-674986770835-kube-api-access-jwxxn\") pod \"7d04f9e5-4726-443a-bf43-674986770835\" (UID: \"7d04f9e5-4726-443a-bf43-674986770835\") "
Oct 30 21:38:12.963603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:12.963291    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d04f9e5-4726-443a-bf43-674986770835-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d04f9e5-4726-443a-bf43-674986770835-maas-api-xks-ca\") pod \"7d04f9e5-4726-443a-bf43-674986770835\" (UID: \"7d04f9e5-4726-443a-bf43-674986770835\") "
Oct 30 21:38:12.963603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:12.963519    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d04f9e5-4726-443a-bf43-674986770835-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d04f9e5-4726-443a-bf43-674986770835-maas-api-openshift-ca\") pod \"7d04f9e5-4726-443a-bf43-674986770835\" (UID: \"7d04f9e5-4726-443a-bf43-674986770835\") "
Oct 30 21:38:12.963919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:12.963847    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d04f9e5-4726-443a-bf43-674986770835-maas-api-openshift-ca" pod "7d04f9e5-4726-443a-bf43-674986770835" (UID: "7d04f9e5-4726-443a-bf43-674986770835"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:12.964631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:12.964610    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d04f9e5-4726-443a-bf43-674986770835-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:12.965874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:12.965848    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d04f9e5-4726-443a-bf43-674986770835-kube-api-access-jwxxn" pod "7d04f9e5-4726-443a-bf43-674986770835" (UID: "7d04f9e5-4726-443a-bf43-674986770835"). InnerVolumeSpecName "kube-api-access-jwxxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:12.966091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:12.966067    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d04f9e5-4726-443a-bf43-674986770835-maas-api-xks-ca" pod "7d04f9e5-4726-443a-bf43-674986770835" (UID: "7d04f9e5-4726-443a-bf43-674986770835"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:13.066625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:13.066587    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwxxn\" (UniqueName: \"kubernetes.io/projected/7d04f9e5-4726-443a-bf43-674986770835-kube-api-access-jwxxn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:13.066625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:13.066620    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d04f9e5-4726-443a-bf43-674986770835-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:13.711899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:13.711868    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"
Oct 30 21:38:13.712083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:13.711905    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7" event={"ID":"7d04f9e5-4726-443a-bf43-674986770835","Type":"ContainerDied","Data":"a980824901cf9fb128f4e2ba7c4217e95c9aab1f2b6e277a0f1bfd383158f36c"}
Oct 30 21:38:13.712083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:13.711958    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a980824901cf9fb128f4e2ba7c4217e95c9aab1f2b6e277a0f1bfd383158f36c"
Oct 30 21:38:18.618543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:18.618515    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:38:18.692468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:18.692433    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"]
Oct 30 21:38:23.144106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.144073    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"]
Oct 30 21:38:23.145038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.145020    4918 cpu_manager.go:414] "containerMap: removing container" podUID="7d04f9e5-4726-443a-bf43-674986770835" containerName="revoke-keys"
Oct 30 21:38:23.145038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.145038    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d04f9e5-4726-443a-bf43-674986770835" containerName="revoke-keys"
Oct 30 21:38:23.145168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.145159    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d04f9e5-4726-443a-bf43-674986770835" containerName="revoke-keys"
Oct 30 21:38:23.149233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.149217    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:23.154705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.154681    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:38:23.155355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.155335    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:38:23.155765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.155735    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:38:23.155972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.155949    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:38:23.159808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.159788    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"]
Oct 30 21:38:23.228196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.228155    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31445c7e-gvs94\" (UID: \"4789b259-18ec-40bc-a923-41f73bbc7d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:23.228377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.228283    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5njm\" (UniqueName: \"kubernetes.io/projected/4789b259-18ec-40bc-a923-41f73bbc7d43-kube-api-access-q5njm\") pod \"maas-api-revoke-sub-31445c7e-gvs94\" (UID: \"4789b259-18ec-40bc-a923-41f73bbc7d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:23.228377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.228342    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31445c7e-gvs94\" (UID: \"4789b259-18ec-40bc-a923-41f73bbc7d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:23.330229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.330193    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31445c7e-gvs94\" (UID: \"4789b259-18ec-40bc-a923-41f73bbc7d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:23.330476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.330260    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5njm\" (UniqueName: \"kubernetes.io/projected/4789b259-18ec-40bc-a923-41f73bbc7d43-kube-api-access-q5njm\") pod \"maas-api-revoke-sub-31445c7e-gvs94\" (UID: \"4789b259-18ec-40bc-a923-41f73bbc7d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:23.330476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.330307    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31445c7e-gvs94\" (UID: \"4789b259-18ec-40bc-a923-41f73bbc7d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:23.330946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.330921    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31445c7e-gvs94\" (UID: \"4789b259-18ec-40bc-a923-41f73bbc7d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:23.332935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.332916    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31445c7e-gvs94\" (UID: \"4789b259-18ec-40bc-a923-41f73bbc7d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:23.340373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.340354    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5njm\" (UniqueName: \"kubernetes.io/projected/4789b259-18ec-40bc-a923-41f73bbc7d43-kube-api-access-q5njm\") pod \"maas-api-revoke-sub-31445c7e-gvs94\" (UID: \"4789b259-18ec-40bc-a923-41f73bbc7d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:23.475467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.473951    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:23.692969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.692924    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw" podUID="ca23f53e-357d-4de2-a35a-c7e882beef6b" containerName="payload-processing" containerID="cri-o://2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a" gracePeriod=25
Oct 30 21:38:23.811758 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:38:23.811728    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4789b259_18ec_40bc_a923_41f73bbc7d43.slice/crio-9b0622e890691f42b6b8345789ed1889d3a8d64730c016c42fd3e74c9f1d30e7 WatchSource:0}: Error finding container 9b0622e890691f42b6b8345789ed1889d3a8d64730c016c42fd3e74c9f1d30e7: Status 404 returned error can't find the container with id 9b0622e890691f42b6b8345789ed1889d3a8d64730c016c42fd3e74c9f1d30e7
Oct 30 21:38:23.812022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.812000    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"]
Oct 30 21:38:23.986223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:23.986163    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:38:24.042254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.042215    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-routing-overlay-volume\") pod \"ca23f53e-357d-4de2-a35a-c7e882beef6b\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") "
Oct 30 21:38:24.042481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.042461    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-provider-credentials\") pod \"ca23f53e-357d-4de2-a35a-c7e882beef6b\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") "
Oct 30 21:38:24.042611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.042586    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-kube-api-access-l6mz6\" (UniqueName: \"kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-kube-api-access-l6mz6\") pod \"ca23f53e-357d-4de2-a35a-c7e882beef6b\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") "
Oct 30 21:38:24.042730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.042692    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-routing-overlay-volume" pod "ca23f53e-357d-4de2-a35a-c7e882beef6b" (UID: "ca23f53e-357d-4de2-a35a-c7e882beef6b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:24.042819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.042726    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-plugins-config-volume\") pod \"ca23f53e-357d-4de2-a35a-c7e882beef6b\" (UID: \"ca23f53e-357d-4de2-a35a-c7e882beef6b\") "
Oct 30 21:38:24.043241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.043210    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-plugins-config-volume" pod "ca23f53e-357d-4de2-a35a-c7e882beef6b" (UID: "ca23f53e-357d-4de2-a35a-c7e882beef6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:24.043936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.043906    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:24.044022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.043941    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca23f53e-357d-4de2-a35a-c7e882beef6b-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:24.045445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.045404    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-kube-api-access-l6mz6" pod "ca23f53e-357d-4de2-a35a-c7e882beef6b" (UID: "ca23f53e-357d-4de2-a35a-c7e882beef6b"). InnerVolumeSpecName "kube-api-access-l6mz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:24.045542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.045504    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-provider-credentials" pod "ca23f53e-357d-4de2-a35a-c7e882beef6b" (UID: "ca23f53e-357d-4de2-a35a-c7e882beef6b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:24.145663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.145629    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:24.145663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.145656    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6mz6\" (UniqueName: \"kubernetes.io/projected/ca23f53e-357d-4de2-a35a-c7e882beef6b-kube-api-access-l6mz6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:24.810254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.810220    4918 generic.go:356] "Generic (PLEG): container finished" podID="ca23f53e-357d-4de2-a35a-c7e882beef6b" containerID="2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a" exitCode=0
Oct 30 21:38:24.810482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.810269    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"
Oct 30 21:38:24.810482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.810303    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw" event={"ID":"ca23f53e-357d-4de2-a35a-c7e882beef6b","Type":"ContainerDied","Data":"2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a"}
Oct 30 21:38:24.810482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.810372    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw" event={"ID":"ca23f53e-357d-4de2-a35a-c7e882beef6b","Type":"ContainerDied","Data":"35b4bd96ce5002433b15e9068aac8961a7f8f42a06a3365fc08149d582c85bed"}
Oct 30 21:38:24.810482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.810442    4918 scope.go:122] "RemoveContainer" containerID="2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a"
Oct 30 21:38:24.812319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.812295    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94" event={"ID":"4789b259-18ec-40bc-a923-41f73bbc7d43","Type":"ContainerStarted","Data":"f4b93013abf0fd76fead34cac8274b928e7f341bd8daa13b172bd7835584aae1"}
Oct 30 21:38:24.812486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.812340    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94" event={"ID":"4789b259-18ec-40bc-a923-41f73bbc7d43","Type":"ContainerStarted","Data":"9b0622e890691f42b6b8345789ed1889d3a8d64730c016c42fd3e74c9f1d30e7"}
Oct 30 21:38:24.825200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.825184    4918 scope.go:122] "RemoveContainer" containerID="2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a"
Oct 30 21:38:24.825522 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:38:24.825492    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a\": container with ID starting with 2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a not found: ID does not exist" containerID="2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a"
Oct 30 21:38:24.825592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.825529    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a"} err="failed to get container status \"2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a\": rpc error: code = NotFound desc = could not find container \"2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a\": container with ID starting with 2489f16e3d1ae4fa298835192edbce08e5b4602ae86c52282f7a2b5b6139897a not found: ID does not exist"
Oct 30 21:38:24.834555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.834518    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94" podStartSLOduration=1.8345068709999999 podStartE2EDuration="1.834506871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:24.831541791 +0000 UTC m=+2217.878100955" watchObservedRunningTime="2026-10-30 21:38:24.834506871 +0000 UTC m=+2217.881066016"
Oct 30 21:38:24.853525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.853499    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"]
Oct 30 21:38:24.858645 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:24.858621    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588lqdfw"]
Oct 30 21:38:25.084199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:25.084124    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca23f53e-357d-4de2-a35a-c7e882beef6b" path="/var/lib/kubelet/pods/ca23f53e-357d-4de2-a35a-c7e882beef6b/volumes"
Oct 30 21:38:25.826026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:25.825997    4918 generic.go:356] "Generic (PLEG): container finished" podID="4789b259-18ec-40bc-a923-41f73bbc7d43" containerID="f4b93013abf0fd76fead34cac8274b928e7f341bd8daa13b172bd7835584aae1" exitCode=0
Oct 30 21:38:25.826451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:25.826074    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94" event={"ID":"4789b259-18ec-40bc-a923-41f73bbc7d43","Type":"ContainerDied","Data":"f4b93013abf0fd76fead34cac8274b928e7f341bd8daa13b172bd7835584aae1"}
Oct 30 21:38:27.150542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.150482    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:27.195010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.194974    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4789b259-18ec-40bc-a923-41f73bbc7d43-kube-api-access-q5njm\" (UniqueName: \"kubernetes.io/projected/4789b259-18ec-40bc-a923-41f73bbc7d43-kube-api-access-q5njm\") pod \"4789b259-18ec-40bc-a923-41f73bbc7d43\" (UID: \"4789b259-18ec-40bc-a923-41f73bbc7d43\") "
Oct 30 21:38:27.195223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.195212    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-xks-ca\") pod \"4789b259-18ec-40bc-a923-41f73bbc7d43\" (UID: \"4789b259-18ec-40bc-a923-41f73bbc7d43\") "
Oct 30 21:38:27.195435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.195358    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-openshift-ca\") pod \"4789b259-18ec-40bc-a923-41f73bbc7d43\" (UID: \"4789b259-18ec-40bc-a923-41f73bbc7d43\") "
Oct 30 21:38:27.196437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.196158    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-openshift-ca" pod "4789b259-18ec-40bc-a923-41f73bbc7d43" (UID: "4789b259-18ec-40bc-a923-41f73bbc7d43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:27.201437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.199159    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4789b259-18ec-40bc-a923-41f73bbc7d43-kube-api-access-q5njm" pod "4789b259-18ec-40bc-a923-41f73bbc7d43" (UID: "4789b259-18ec-40bc-a923-41f73bbc7d43"). InnerVolumeSpecName "kube-api-access-q5njm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:27.201437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.199932    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-xks-ca" pod "4789b259-18ec-40bc-a923-41f73bbc7d43" (UID: "4789b259-18ec-40bc-a923-41f73bbc7d43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:27.297428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.297383    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:27.297428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.297426    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5njm\" (UniqueName: \"kubernetes.io/projected/4789b259-18ec-40bc-a923-41f73bbc7d43-kube-api-access-q5njm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:27.297428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.297436    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4789b259-18ec-40bc-a923-41f73bbc7d43-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:27.844125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.844093    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:38:27.844309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.844143    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94" event={"ID":"4789b259-18ec-40bc-a923-41f73bbc7d43","Type":"ContainerDied","Data":"9b0622e890691f42b6b8345789ed1889d3a8d64730c016c42fd3e74c9f1d30e7"}
Oct 30 21:38:27.844309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:27.844190    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b0622e890691f42b6b8345789ed1889d3a8d64730c016c42fd3e74c9f1d30e7"
Oct 30 21:38:29.209593 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:38:29.194642    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:38:32.024121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.024091    4918 scope.go:122] "RemoveContainer" containerID="36af050bba971af0c5bf48eb78dff42ded839ab2a2007b58ebf5b00203af28af"
Oct 30 21:38:32.037087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.037059    4918 scope.go:122] "RemoveContainer" containerID="d0a0bbdbca96b709ed1efaa0193522f16709aee2f795580bfe252636edd12194"
Oct 30 21:38:32.048347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.048323    4918 scope.go:122] "RemoveContainer" containerID="dd8a1e3a7506d20bce298c844be5acaa3c3ca5754475bd65a8879959e77ecad6"
Oct 30 21:38:32.059485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.059459    4918 scope.go:122] "RemoveContainer" containerID="a3c63b34f59d3cf3c65e34ad14da3ee2c066f29d6221210035ec5bba49836aa3"
Oct 30 21:38:32.733824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.733787    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"]
Oct 30 21:38:32.734639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.734623    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ca23f53e-357d-4de2-a35a-c7e882beef6b" containerName="payload-processing"
Oct 30 21:38:32.734693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.734641    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca23f53e-357d-4de2-a35a-c7e882beef6b" containerName="payload-processing"
Oct 30 21:38:32.734693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.734682    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4789b259-18ec-40bc-a923-41f73bbc7d43" containerName="revoke-keys"
Oct 30 21:38:32.734693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.734687    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4789b259-18ec-40bc-a923-41f73bbc7d43" containerName="revoke-keys"
Oct 30 21:38:32.734833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.734824    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4789b259-18ec-40bc-a923-41f73bbc7d43" containerName="revoke-keys"
Oct 30 21:38:32.734870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.734843    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca23f53e-357d-4de2-a35a-c7e882beef6b" containerName="payload-processing"
Oct 30 21:38:32.740976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.740955    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:32.746564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.746538    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f333b4-kgp57\" (UID: \"31454eb8-a101-4a6e-aedc-6e8d5588a827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:32.746702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.746679    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f333b4-kgp57\" (UID: \"31454eb8-a101-4a6e-aedc-6e8d5588a827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:32.746884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.746862    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzfzt\" (UniqueName: \"kubernetes.io/projected/31454eb8-a101-4a6e-aedc-6e8d5588a827-kube-api-access-qzfzt\") pod \"maas-api-revoke-sub-e4f333b4-kgp57\" (UID: \"31454eb8-a101-4a6e-aedc-6e8d5588a827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:32.748528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.748509    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:38:32.748718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.748699    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:38:32.748718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.748722    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:38:32.748914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.748746    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:38:32.749659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.749638    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"]
Oct 30 21:38:32.869094 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.861848    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f333b4-kgp57\" (UID: \"31454eb8-a101-4a6e-aedc-6e8d5588a827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:32.869094 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.862069    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzfzt\" (UniqueName: \"kubernetes.io/projected/31454eb8-a101-4a6e-aedc-6e8d5588a827-kube-api-access-qzfzt\") pod \"maas-api-revoke-sub-e4f333b4-kgp57\" (UID: \"31454eb8-a101-4a6e-aedc-6e8d5588a827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:32.869094 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.862466    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f333b4-kgp57\" (UID: \"31454eb8-a101-4a6e-aedc-6e8d5588a827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:32.879573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.879421    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzfzt\" (UniqueName: \"kubernetes.io/projected/31454eb8-a101-4a6e-aedc-6e8d5588a827-kube-api-access-qzfzt\") pod \"maas-api-revoke-sub-e4f333b4-kgp57\" (UID: \"31454eb8-a101-4a6e-aedc-6e8d5588a827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:32.887525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.881018    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4f333b4-kgp57\" (UID: \"31454eb8-a101-4a6e-aedc-6e8d5588a827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:32.902759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:32.902732    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4f333b4-kgp57\" (UID: \"31454eb8-a101-4a6e-aedc-6e8d5588a827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:33.057585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:33.057499    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:33.347917 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:38:33.347891    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31454eb8_a101_4a6e_aedc_6e8d5588a827.slice/crio-d94d9dfb4c30407f405a507eed11d169cab6e411ecbc9a1f29a6268d7f983cc1 WatchSource:0}: Error finding container d94d9dfb4c30407f405a507eed11d169cab6e411ecbc9a1f29a6268d7f983cc1: Status 404 returned error can't find the container with id d94d9dfb4c30407f405a507eed11d169cab6e411ecbc9a1f29a6268d7f983cc1
Oct 30 21:38:33.349902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:33.349874    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"]
Oct 30 21:38:34.047109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:34.047070    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57" event={"ID":"31454eb8-a101-4a6e-aedc-6e8d5588a827","Type":"ContainerStarted","Data":"973332ba009b96ecea2510c0343e179e2563ef7151814c36daabd6e8d25e59d7"}
Oct 30 21:38:34.047299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:34.047134    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57" event={"ID":"31454eb8-a101-4a6e-aedc-6e8d5588a827","Type":"ContainerStarted","Data":"d94d9dfb4c30407f405a507eed11d169cab6e411ecbc9a1f29a6268d7f983cc1"}
Oct 30 21:38:34.065793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:34.065746    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57" podStartSLOduration=2.0657317060000002 podStartE2EDuration="2.065731706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:34.063504878 +0000 UTC m=+2227.110064024" watchObservedRunningTime="2026-10-30 21:38:34.065731706 +0000 UTC m=+2227.112290851"
Oct 30 21:38:34.299258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:34.296716    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"]
Oct 30 21:38:35.057846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:35.057805    4918 generic.go:356] "Generic (PLEG): container finished" podID="31454eb8-a101-4a6e-aedc-6e8d5588a827" containerID="973332ba009b96ecea2510c0343e179e2563ef7151814c36daabd6e8d25e59d7" exitCode=0
Oct 30 21:38:35.058038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:35.057877    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57" event={"ID":"31454eb8-a101-4a6e-aedc-6e8d5588a827","Type":"ContainerDied","Data":"973332ba009b96ecea2510c0343e179e2563ef7151814c36daabd6e8d25e59d7"}
Oct 30 21:38:36.350093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:36.350065    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:36.497442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:36.497372    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-openshift-ca\") pod \"31454eb8-a101-4a6e-aedc-6e8d5588a827\" (UID: \"31454eb8-a101-4a6e-aedc-6e8d5588a827\") "
Oct 30 21:38:36.497617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:36.497578    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31454eb8-a101-4a6e-aedc-6e8d5588a827-kube-api-access-qzfzt\" (UniqueName: \"kubernetes.io/projected/31454eb8-a101-4a6e-aedc-6e8d5588a827-kube-api-access-qzfzt\") pod \"31454eb8-a101-4a6e-aedc-6e8d5588a827\" (UID: \"31454eb8-a101-4a6e-aedc-6e8d5588a827\") "
Oct 30 21:38:36.497687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:36.497644    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-xks-ca\") pod \"31454eb8-a101-4a6e-aedc-6e8d5588a827\" (UID: \"31454eb8-a101-4a6e-aedc-6e8d5588a827\") "
Oct 30 21:38:36.497744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:36.497686    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-openshift-ca" pod "31454eb8-a101-4a6e-aedc-6e8d5588a827" (UID: "31454eb8-a101-4a6e-aedc-6e8d5588a827"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:36.498233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:36.498216    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:36.499789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:36.499759    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-xks-ca" pod "31454eb8-a101-4a6e-aedc-6e8d5588a827" (UID: "31454eb8-a101-4a6e-aedc-6e8d5588a827"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:36.499884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:36.499832    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31454eb8-a101-4a6e-aedc-6e8d5588a827-kube-api-access-qzfzt" pod "31454eb8-a101-4a6e-aedc-6e8d5588a827" (UID: "31454eb8-a101-4a6e-aedc-6e8d5588a827"). InnerVolumeSpecName "kube-api-access-qzfzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:36.599642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:36.599575    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzfzt\" (UniqueName: \"kubernetes.io/projected/31454eb8-a101-4a6e-aedc-6e8d5588a827-kube-api-access-qzfzt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:36.599642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:36.599599    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31454eb8-a101-4a6e-aedc-6e8d5588a827-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:37.093784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:37.093756    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"
Oct 30 21:38:37.094016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:37.093795    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57" event={"ID":"31454eb8-a101-4a6e-aedc-6e8d5588a827","Type":"ContainerDied","Data":"d94d9dfb4c30407f405a507eed11d169cab6e411ecbc9a1f29a6268d7f983cc1"}
Oct 30 21:38:37.094016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:37.093849    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d94d9dfb4c30407f405a507eed11d169cab6e411ecbc9a1f29a6268d7f983cc1"
Oct 30 21:38:39.297302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.297256    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq" podUID="cc4e1a61-e664-47f4-8b36-307285aba36d" containerName="payload-processing" containerID="cri-o://595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2" gracePeriod=25
Oct 30 21:38:39.590591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.590567    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:38:39.670981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.670944    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-routing-overlay-volume\") pod \"cc4e1a61-e664-47f4-8b36-307285aba36d\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") "
Oct 30 21:38:39.671227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.671201    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-plugins-config-volume\") pod \"cc4e1a61-e664-47f4-8b36-307285aba36d\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") "
Oct 30 21:38:39.671375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.671320    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-kube-api-access-mq92w\" (UniqueName: \"kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-kube-api-access-mq92w\") pod \"cc4e1a61-e664-47f4-8b36-307285aba36d\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") "
Oct 30 21:38:39.671375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.671324    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-routing-overlay-volume" pod "cc4e1a61-e664-47f4-8b36-307285aba36d" (UID: "cc4e1a61-e664-47f4-8b36-307285aba36d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:39.671551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.671411    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-provider-credentials\") pod \"cc4e1a61-e664-47f4-8b36-307285aba36d\" (UID: \"cc4e1a61-e664-47f4-8b36-307285aba36d\") "
Oct 30 21:38:39.671737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.671708    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-plugins-config-volume" pod "cc4e1a61-e664-47f4-8b36-307285aba36d" (UID: "cc4e1a61-e664-47f4-8b36-307285aba36d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:39.672355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.672331    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:39.672492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.672361    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc4e1a61-e664-47f4-8b36-307285aba36d-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:39.673847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.673824    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-provider-credentials" pod "cc4e1a61-e664-47f4-8b36-307285aba36d" (UID: "cc4e1a61-e664-47f4-8b36-307285aba36d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:39.674351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.674332    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-kube-api-access-mq92w" pod "cc4e1a61-e664-47f4-8b36-307285aba36d" (UID: "cc4e1a61-e664-47f4-8b36-307285aba36d"). InnerVolumeSpecName "kube-api-access-mq92w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:39.774431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.774403    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq92w\" (UniqueName: \"kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-kube-api-access-mq92w\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:39.774431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:39.774429    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc4e1a61-e664-47f4-8b36-307285aba36d-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:40.120268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:40.120237    4918 generic.go:356] "Generic (PLEG): container finished" podID="cc4e1a61-e664-47f4-8b36-307285aba36d" containerID="595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2" exitCode=0
Oct 30 21:38:40.120482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:40.120296    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"
Oct 30 21:38:40.120482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:40.120337    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq" event={"ID":"cc4e1a61-e664-47f4-8b36-307285aba36d","Type":"ContainerDied","Data":"595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2"}
Oct 30 21:38:40.120482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:40.120423    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq" event={"ID":"cc4e1a61-e664-47f4-8b36-307285aba36d","Type":"ContainerDied","Data":"4ea7438bf4a5b71ec6351a549d8d84d13b3f269b9c6c32db0a4ad1e419369cdf"}
Oct 30 21:38:40.120482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:40.120459    4918 scope.go:122] "RemoveContainer" containerID="595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2"
Oct 30 21:38:40.133554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:40.133534    4918 scope.go:122] "RemoveContainer" containerID="595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2"
Oct 30 21:38:40.133895 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:38:40.133870    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2\": container with ID starting with 595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2 not found: ID does not exist" containerID="595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2"
Oct 30 21:38:40.133991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:40.133906    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2"} err="failed to get container status \"595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2\": rpc error: code = NotFound desc = could not find container \"595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2\": container with ID starting with 595d73c0842af6167c76e2079791a66e763fe63c9f89ab84ead5af5f0869f7d2 not found: ID does not exist"
Oct 30 21:38:40.147756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:40.147721    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"]
Oct 30 21:38:40.152590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:40.152568    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-b754nc4jq"]
Oct 30 21:38:41.082264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:41.082227    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4e1a61-e664-47f4-8b36-307285aba36d" path="/var/lib/kubelet/pods/cc4e1a61-e664-47f4-8b36-307285aba36d/volumes"
Oct 30 21:38:43.984419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:43.979284    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"]
Oct 30 21:38:43.984419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:43.980647    4918 cpu_manager.go:414] "containerMap: removing container" podUID="31454eb8-a101-4a6e-aedc-6e8d5588a827" containerName="revoke-keys"
Oct 30 21:38:43.984419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:43.980669    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31454eb8-a101-4a6e-aedc-6e8d5588a827" containerName="revoke-keys"
Oct 30 21:38:43.984419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:43.980724    4918 cpu_manager.go:414] "containerMap: removing container" podUID="cc4e1a61-e664-47f4-8b36-307285aba36d" containerName="payload-processing"
Oct 30 21:38:43.984419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:43.980732    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4e1a61-e664-47f4-8b36-307285aba36d" containerName="payload-processing"
Oct 30 21:38:43.984419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:43.980937    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4e1a61-e664-47f4-8b36-307285aba36d" containerName="payload-processing"
Oct 30 21:38:43.984419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:43.980962    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="31454eb8-a101-4a6e-aedc-6e8d5588a827" containerName="revoke-keys"
Oct 30 21:38:44.157161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.157128    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.165381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.165351    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"]
Oct 30 21:38:44.373476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.370649    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5f9g\" (UniqueName: \"kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-kube-api-access-p5f9g\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.373476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.370733    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.373476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.371013    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.373476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.371086    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.555809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.555766    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5f9g\" (UniqueName: \"kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-kube-api-access-p5f9g\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.556103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.556084    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.556542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.556516    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.556742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.556726    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.567412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.557855    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.567677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.559073    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.580337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.579726    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5f9g\" (UniqueName: \"kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-kube-api-access-p5f9g\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.587521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.585654    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"]
Oct 30 21:38:44.647472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.618894    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.670029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.669997    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"]
Oct 30 21:38:44.670373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.670352    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:44.681245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.681218    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:38:44.681585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.681568    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:38:44.681746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.681733    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:38:44.682556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.682537    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:38:44.774861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.774812    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ecda13-44be-4b82-9719-04ef06924244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c9f1e32-6ksdg\" (UID: \"08ecda13-44be-4b82-9719-04ef06924244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:44.775065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.774881    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz2hr\" (UniqueName: \"kubernetes.io/projected/08ecda13-44be-4b82-9719-04ef06924244-kube-api-access-gz2hr\") pod \"maas-api-revoke-sub-8c9f1e32-6ksdg\" (UID: \"08ecda13-44be-4b82-9719-04ef06924244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:44.775065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.775019    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ecda13-44be-4b82-9719-04ef06924244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c9f1e32-6ksdg\" (UID: \"08ecda13-44be-4b82-9719-04ef06924244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:44.783684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.783651    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:44.878490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.878060    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ecda13-44be-4b82-9719-04ef06924244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c9f1e32-6ksdg\" (UID: \"08ecda13-44be-4b82-9719-04ef06924244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:44.878490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.878115    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz2hr\" (UniqueName: \"kubernetes.io/projected/08ecda13-44be-4b82-9719-04ef06924244-kube-api-access-gz2hr\") pod \"maas-api-revoke-sub-8c9f1e32-6ksdg\" (UID: \"08ecda13-44be-4b82-9719-04ef06924244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:44.881197 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.880897    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ecda13-44be-4b82-9719-04ef06924244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c9f1e32-6ksdg\" (UID: \"08ecda13-44be-4b82-9719-04ef06924244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:44.882559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.881758    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ecda13-44be-4b82-9719-04ef06924244-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c9f1e32-6ksdg\" (UID: \"08ecda13-44be-4b82-9719-04ef06924244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:44.888745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.888695    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ecda13-44be-4b82-9719-04ef06924244-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c9f1e32-6ksdg\" (UID: \"08ecda13-44be-4b82-9719-04ef06924244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:44.909025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:44.908946    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz2hr\" (UniqueName: \"kubernetes.io/projected/08ecda13-44be-4b82-9719-04ef06924244-kube-api-access-gz2hr\") pod \"maas-api-revoke-sub-8c9f1e32-6ksdg\" (UID: \"08ecda13-44be-4b82-9719-04ef06924244\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:45.015814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:45.015781    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:45.086656 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:38:45.086624    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924e845c_e3ad_4ac8_9991_24e5fb40b7b8.slice/crio-5fe5d15c8e6363fb5d374174bc550595224af74fb29c330b65757dac9b3c61a4 WatchSource:0}: Error finding container 5fe5d15c8e6363fb5d374174bc550595224af74fb29c330b65757dac9b3c61a4: Status 404 returned error can't find the container with id 5fe5d15c8e6363fb5d374174bc550595224af74fb29c330b65757dac9b3c61a4
Oct 30 21:38:45.092599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:45.089836    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"]
Oct 30 21:38:45.298564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:45.298471    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2" event={"ID":"924e845c-e3ad-4ac8-9991-24e5fb40b7b8","Type":"ContainerStarted","Data":"5fe5d15c8e6363fb5d374174bc550595224af74fb29c330b65757dac9b3c61a4"}
Oct 30 21:38:45.367742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:45.367709    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"]
Oct 30 21:38:46.312526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:46.312483    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg" event={"ID":"08ecda13-44be-4b82-9719-04ef06924244","Type":"ContainerStarted","Data":"16abb8bdd03f247ec46569dc4f813ec4a3645f941cf9cb641dbebdfacfa4041c"}
Oct 30 21:38:46.312986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:46.312556    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg" event={"ID":"08ecda13-44be-4b82-9719-04ef06924244","Type":"ContainerStarted","Data":"86c8595dbfc0c9609feb440272c7f92a351c982497b04b5e776286516a538bd1"}
Oct 30 21:38:46.314362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:46.314342    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2" event={"ID":"924e845c-e3ad-4ac8-9991-24e5fb40b7b8","Type":"ContainerStarted","Data":"6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f"}
Oct 30 21:38:46.314481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:46.314432    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:46.348884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:46.348835    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2" podStartSLOduration=3.348822563 podStartE2EDuration="3.348822563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:46.344980739 +0000 UTC m=+2239.391539887" watchObservedRunningTime="2026-10-30 21:38:46.348822563 +0000 UTC m=+2239.395381708"
Oct 30 21:38:47.324341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:47.324302    4918 generic.go:356] "Generic (PLEG): container finished" podID="08ecda13-44be-4b82-9719-04ef06924244" containerID="16abb8bdd03f247ec46569dc4f813ec4a3645f941cf9cb641dbebdfacfa4041c" exitCode=0
Oct 30 21:38:47.324772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:47.324423    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg" event={"ID":"08ecda13-44be-4b82-9719-04ef06924244","Type":"ContainerDied","Data":"16abb8bdd03f247ec46569dc4f813ec4a3645f941cf9cb641dbebdfacfa4041c"}
Oct 30 21:38:47.613297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:47.613273    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:47.768523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:47.768492    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ecda13-44be-4b82-9719-04ef06924244-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ecda13-44be-4b82-9719-04ef06924244-maas-api-openshift-ca\") pod \"08ecda13-44be-4b82-9719-04ef06924244\" (UID: \"08ecda13-44be-4b82-9719-04ef06924244\") "
Oct 30 21:38:47.768693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:47.768672    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ecda13-44be-4b82-9719-04ef06924244-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ecda13-44be-4b82-9719-04ef06924244-maas-api-xks-ca\") pod \"08ecda13-44be-4b82-9719-04ef06924244\" (UID: \"08ecda13-44be-4b82-9719-04ef06924244\") "
Oct 30 21:38:47.768805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:47.768789    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ecda13-44be-4b82-9719-04ef06924244-kube-api-access-gz2hr\" (UniqueName: \"kubernetes.io/projected/08ecda13-44be-4b82-9719-04ef06924244-kube-api-access-gz2hr\") pod \"08ecda13-44be-4b82-9719-04ef06924244\" (UID: \"08ecda13-44be-4b82-9719-04ef06924244\") "
Oct 30 21:38:47.768895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:47.768871    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ecda13-44be-4b82-9719-04ef06924244-maas-api-openshift-ca" pod "08ecda13-44be-4b82-9719-04ef06924244" (UID: "08ecda13-44be-4b82-9719-04ef06924244"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:47.769366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:47.769292    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ecda13-44be-4b82-9719-04ef06924244-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:47.770932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:47.770905    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ecda13-44be-4b82-9719-04ef06924244-maas-api-xks-ca" pod "08ecda13-44be-4b82-9719-04ef06924244" (UID: "08ecda13-44be-4b82-9719-04ef06924244"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:47.771024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:47.770984    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ecda13-44be-4b82-9719-04ef06924244-kube-api-access-gz2hr" pod "08ecda13-44be-4b82-9719-04ef06924244" (UID: "08ecda13-44be-4b82-9719-04ef06924244"). InnerVolumeSpecName "kube-api-access-gz2hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:47.870558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:47.870488    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz2hr\" (UniqueName: \"kubernetes.io/projected/08ecda13-44be-4b82-9719-04ef06924244-kube-api-access-gz2hr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:47.870558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:47.870517    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ecda13-44be-4b82-9719-04ef06924244-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:48.336483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:48.336449    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg" event={"ID":"08ecda13-44be-4b82-9719-04ef06924244","Type":"ContainerDied","Data":"86c8595dbfc0c9609feb440272c7f92a351c982497b04b5e776286516a538bd1"}
Oct 30 21:38:48.336483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:48.336471    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"
Oct 30 21:38:48.336925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:48.336509    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86c8595dbfc0c9609feb440272c7f92a351c982497b04b5e776286516a538bd1"
Oct 30 21:38:50.155112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.155076    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"]
Oct 30 21:38:50.155939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.155923    4918 cpu_manager.go:414] "containerMap: removing container" podUID="08ecda13-44be-4b82-9719-04ef06924244" containerName="revoke-keys"
Oct 30 21:38:50.155988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.155941    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ecda13-44be-4b82-9719-04ef06924244" containerName="revoke-keys"
Oct 30 21:38:50.156111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.156102    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ecda13-44be-4b82-9719-04ef06924244" containerName="revoke-keys"
Oct 30 21:38:50.160154 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.160137    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.170417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.170368    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"openshift-service-ca.crt\""
Oct 30 21:38:50.170548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.170442    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"agc-cred-64ea8ebbfd9e\""
Oct 30 21:38:50.170548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.170443    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e596df\""
Oct 30 21:38:50.170548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.170531    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"kube-root-ca.crt\""
Oct 30 21:38:50.170679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.170569    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"routing-overlay\""
Oct 30 21:38:50.170679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.170624    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"payload-processing-external-model-e2e--57c066ea-dockercfg-nwq84\""
Oct 30 21:38:50.175238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.175219    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"]
Oct 30 21:38:50.289492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.267376    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.290247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.290209    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.326433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.290792    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.326846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.326809    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8ch5\" (UniqueName: \"kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-kube-api-access-n8ch5\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.434474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.433989    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.434474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.434063    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ch5\" (UniqueName: \"kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-kube-api-access-n8ch5\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.434474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.434134    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.434474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.434286    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.445413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.435297    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.446551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.446515    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.474416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.470567    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8ch5\" (UniqueName: \"kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-kube-api-access-n8ch5\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.482421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.481886    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.482421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.482182    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:50.613612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.613576    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"]
Oct 30 21:38:50.631482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.631453    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:50.663096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.663048    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"agc-cred-64a68af6a9b2\""
Oct 30 21:38:50.709125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.709088    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"]
Oct 30 21:38:50.742958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.742862    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:50.742958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.742903    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kxfs\" (UniqueName: \"kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-kube-api-access-9kxfs\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:50.743229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.743023    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:50.743473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.743438    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:50.847412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.845961    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:50.847412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.846133    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:50.847412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.846164    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kxfs\" (UniqueName: \"kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-kube-api-access-9kxfs\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:50.847412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.846257    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:50.951469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.936415    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:50.992454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:50.991487    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:51.154444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:51.151926    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:51.205470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:51.170102    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kxfs\" (UniqueName: \"kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-kube-api-access-9kxfs\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:51.395418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:51.345216    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:51.818524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:51.818066    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"]
Oct 30 21:38:51.845212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:51.838718    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:38:51.874413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:51.870920    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:38:51.980418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:51.980019    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:38:52.021413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.000279    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:38:52.021413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.002581    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:38:52.021413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.004457    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:38:52.031730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.026432    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:38:52.057414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.052793    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:38:52.071466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.065594    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:38:52.094270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.090346    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:38:52.661597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.661484    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:38:52.867868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.861803    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:38:52.867868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.862218    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:38:52.923479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.920417    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv" event={"ID":"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1","Type":"ContainerStarted","Data":"d05f72b7792cee96371da6cb63dafad713452cb41bc01ea70d612f58637740ab"}
Oct 30 21:38:52.923479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.923365    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:38:52.923479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.923423    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv" event={"ID":"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1","Type":"ContainerStarted","Data":"23af41e2a4b6762661967af663154890eb3fb956e481d75cd4819f76ca78a24c"}
Oct 30 21:38:52.928068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.928029    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:38:52.940341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.935261    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:38:52.954224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.954155    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:38:52.984790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.982503    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv" podStartSLOduration=2.982483957 podStartE2EDuration="2.982483957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:52.980074165 +0000 UTC m=+2246.026633311" watchObservedRunningTime="2026-10-30 21:38:52.982483957 +0000 UTC m=+2246.029043106"
Oct 30 21:38:52.986470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:52.985482    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:38:53.305201 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:38:53.305167    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6e64e5_fe59_4338_bbba_7931f806abca.slice/crio-3eb4881d32919e8fb8f2408ac3313c83d866d1a57f4b4a8e04412c30eeb164e9 WatchSource:0}: Error finding container 3eb4881d32919e8fb8f2408ac3313c83d866d1a57f4b4a8e04412c30eeb164e9: Status 404 returned error can't find the container with id 3eb4881d32919e8fb8f2408ac3313c83d866d1a57f4b4a8e04412c30eeb164e9
Oct 30 21:38:53.305986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:53.305958    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"]
Oct 30 21:38:53.933434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:53.933379    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr" event={"ID":"fa6e64e5-fe59-4338-bbba-7931f806abca","Type":"ContainerStarted","Data":"d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a"}
Oct 30 21:38:53.933853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:53.933465    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:38:53.933853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:53.933488    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr" event={"ID":"fa6e64e5-fe59-4338-bbba-7931f806abca","Type":"ContainerStarted","Data":"3eb4881d32919e8fb8f2408ac3313c83d866d1a57f4b4a8e04412c30eeb164e9"}
Oct 30 21:38:53.968424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:53.966220    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr" podStartSLOduration=3.966195956 podStartE2EDuration="3.966195956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:53.963269122 +0000 UTC m=+2247.009828270" watchObservedRunningTime="2026-10-30 21:38:53.966195956 +0000 UTC m=+2247.012755122"
Oct 30 21:38:55.560495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.553586    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"]
Oct 30 21:38:55.607419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.605768    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"]
Oct 30 21:38:55.607419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.605911    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:55.629530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.629450    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"agc-cred-2f614df36ac6\""
Oct 30 21:38:55.633164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.632725    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"openshift-service-ca.crt\""
Oct 30 21:38:55.633164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.632822    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"payload-processing-external-model-e2e--3e617b48-dockercfg-d4sw2\""
Oct 30 21:38:55.633164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.632885    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"agc-cred-2036742396f6\""
Oct 30 21:38:55.633164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.632958    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"routing-overlay\""
Oct 30 21:38:55.633164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.633052    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"kube-root-ca.crt\""
Oct 30 21:38:55.633164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.633110    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-36a099\"/\"payload-processing-external-model-plugins-e2e-extmod-b-36a099\""
Oct 30 21:38:55.766671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.766447    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:55.771444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.771414    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:55.771852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.771825    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pv9g\" (UniqueName: \"kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-kube-api-access-7pv9g\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:55.773296 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.773275    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:55.875781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.875681    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:55.875781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.875723    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:55.876037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.875790    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pv9g\" (UniqueName: \"kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-kube-api-access-7pv9g\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:55.876037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.875917    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:55.876556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.876530    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:55.876677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.876619    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:55.878510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.878483    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:55.884943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:55.884921    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pv9g\" (UniqueName: \"kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-kube-api-access-7pv9g\") pod \"payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") " pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:56.081723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:56.081692    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:56.377774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:56.377736    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"]
Oct 30 21:38:57.107430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.104657    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l" event={"ID":"9daca20b-16ec-4f06-bcb6-ed98ec77a40b","Type":"ContainerStarted","Data":"5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca"}
Oct 30 21:38:57.221439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.213381    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l" podStartSLOduration=2.213357802 podStartE2EDuration="2.213357802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:57.182943094 +0000 UTC m=+2250.229502322" watchObservedRunningTime="2026-10-30 21:38:57.213357802 +0000 UTC m=+2250.259916950"
Oct 30 21:38:57.221439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.215640    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"]
Oct 30 21:38:57.221439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.215706    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l" event={"ID":"9daca20b-16ec-4f06-bcb6-ed98ec77a40b","Type":"ContainerStarted","Data":"e0875c99af0dc85159436b05236abb86a8b509a49e0be9c26d86915c461c6bac"}
Oct 30 21:38:57.221439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.215742    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"]
Oct 30 21:38:57.227243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.227209    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:38:57.227243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.227243    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"]
Oct 30 21:38:57.227493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.227315    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.362090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.361997    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qp88\" (UniqueName: \"kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-kube-api-access-6qp88\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.362265 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.362238    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.362365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.362353    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.362516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.362498    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.486416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.475846    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.486416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.476007    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.486416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.476118    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.486416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.476216    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qp88\" (UniqueName: \"kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-kube-api-access-6qp88\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.486416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.477687    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.486416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.478288    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.524412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.518943    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qp88\" (UniqueName: \"kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-kube-api-access-6qp88\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.554414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.539893    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.554414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.552722    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:57.724120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.722653    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"]
Oct 30 21:38:57.782195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.781734    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:38:57.793491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.792725    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:38:57.793491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.793306    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:38:57.793491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.793414    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:38:57.793491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.793465    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:38:57.797453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.797425    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"]
Oct 30 21:38:57.836306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.836274    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6wjg\" (UniqueName: \"kubernetes.io/projected/03af4727-db4d-40d4-bbc5-8e13561cdf14-kube-api-access-h6wjg\") pod \"maas-api-revoke-sub-e59d8258-cfrl4\" (UID: \"03af4727-db4d-40d4-bbc5-8e13561cdf14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:38:57.836516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.836311    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59d8258-cfrl4\" (UID: \"03af4727-db4d-40d4-bbc5-8e13561cdf14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:38:57.836516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.836377    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59d8258-cfrl4\" (UID: \"03af4727-db4d-40d4-bbc5-8e13561cdf14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:38:57.944458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.943356    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59d8258-cfrl4\" (UID: \"03af4727-db4d-40d4-bbc5-8e13561cdf14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:38:57.946495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.945855    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6wjg\" (UniqueName: \"kubernetes.io/projected/03af4727-db4d-40d4-bbc5-8e13561cdf14-kube-api-access-h6wjg\") pod \"maas-api-revoke-sub-e59d8258-cfrl4\" (UID: \"03af4727-db4d-40d4-bbc5-8e13561cdf14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:38:57.946495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.945923    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59d8258-cfrl4\" (UID: \"03af4727-db4d-40d4-bbc5-8e13561cdf14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:38:57.961427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.955603    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59d8258-cfrl4\" (UID: \"03af4727-db4d-40d4-bbc5-8e13561cdf14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:38:57.967148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.966458    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6wjg\" (UniqueName: \"kubernetes.io/projected/03af4727-db4d-40d4-bbc5-8e13561cdf14-kube-api-access-h6wjg\") pod \"maas-api-revoke-sub-e59d8258-cfrl4\" (UID: \"03af4727-db4d-40d4-bbc5-8e13561cdf14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:38:57.972736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:57.972709    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59d8258-cfrl4\" (UID: \"03af4727-db4d-40d4-bbc5-8e13561cdf14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:38:58.067790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:58.067751    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"]
Oct 30 21:38:58.068241 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:38:58.068215    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3d0b3d_b641_42a0_81af_2643ae1d1b3c.slice/crio-5c68495ad2fca4b2929aab8e6f806eb0d3069f1f6babdd9a84df65a727116f8a WatchSource:0}: Error finding container 5c68495ad2fca4b2929aab8e6f806eb0d3069f1f6babdd9a84df65a727116f8a: Status 404 returned error can't find the container with id 5c68495ad2fca4b2929aab8e6f806eb0d3069f1f6babdd9a84df65a727116f8a
Oct 30 21:38:58.094944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:58.094915    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls" event={"ID":"df3d0b3d-b641-42a0-81af-2643ae1d1b3c","Type":"ContainerStarted","Data":"5c68495ad2fca4b2929aab8e6f806eb0d3069f1f6babdd9a84df65a727116f8a"}
Oct 30 21:38:58.095337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:58.095316    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:38:58.126317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:58.126291    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:38:58.418972 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:38:58.418940    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03af4727_db4d_40d4_bbc5_8e13561cdf14.slice/crio-398d05f4248095fb196b3c2e8a99edaeb37159634036133fa0f176d474e60473 WatchSource:0}: Error finding container 398d05f4248095fb196b3c2e8a99edaeb37159634036133fa0f176d474e60473: Status 404 returned error can't find the container with id 398d05f4248095fb196b3c2e8a99edaeb37159634036133fa0f176d474e60473
Oct 30 21:38:58.419924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:58.419901    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"]
Oct 30 21:38:59.105900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:59.105866    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls" event={"ID":"df3d0b3d-b641-42a0-81af-2643ae1d1b3c","Type":"ContainerStarted","Data":"3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230"}
Oct 30 21:38:59.106096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:59.105937    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:38:59.107872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:59.107841    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4" event={"ID":"03af4727-db4d-40d4-bbc5-8e13561cdf14","Type":"ContainerStarted","Data":"88e71195d6559938922daa37c6525951177a6e35283f6ccafa6d1e4740bd0fcd"}
Oct 30 21:38:59.107994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:59.107892    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4" event={"ID":"03af4727-db4d-40d4-bbc5-8e13561cdf14","Type":"ContainerStarted","Data":"398d05f4248095fb196b3c2e8a99edaeb37159634036133fa0f176d474e60473"}
Oct 30 21:38:59.127073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:59.127007    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls" podStartSLOduration=3.126992952 podStartE2EDuration="3.126992952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:59.12497566 +0000 UTC m=+2252.171534823" watchObservedRunningTime="2026-10-30 21:38:59.126992952 +0000 UTC m=+2252.173552097"
Oct 30 21:38:59.150253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:38:59.150200    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4" podStartSLOduration=2.150186813 podStartE2EDuration="2.150186813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:59.1455726 +0000 UTC m=+2252.192131769" watchObservedRunningTime="2026-10-30 21:38:59.150186813 +0000 UTC m=+2252.196745959"
Oct 30 21:38:59.163784 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:38:59.163710    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:39:00.118268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:00.118233    4918 generic.go:356] "Generic (PLEG): container finished" podID="03af4727-db4d-40d4-bbc5-8e13561cdf14" containerID="88e71195d6559938922daa37c6525951177a6e35283f6ccafa6d1e4740bd0fcd" exitCode=0
Oct 30 21:39:00.118467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:00.118288    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4" event={"ID":"03af4727-db4d-40d4-bbc5-8e13561cdf14","Type":"ContainerDied","Data":"88e71195d6559938922daa37c6525951177a6e35283f6ccafa6d1e4740bd0fcd"}
Oct 30 21:39:00.791574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:00.791545    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"]
Oct 30 21:39:00.797444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:00.797414    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:00.803994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:00.803967    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"]
Oct 30 21:39:00.927907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:00.927872    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d561d836-b7lxc\" (UID: \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:00.928097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:00.928073    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d561d836-b7lxc\" (UID: \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:00.928280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:00.928257    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8882\" (UniqueName: \"kubernetes.io/projected/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-kube-api-access-z8882\") pod \"maas-api-revoke-sub-d561d836-b7lxc\" (UID: \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:01.029752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.029713    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d561d836-b7lxc\" (UID: \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:01.029955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.029851    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d561d836-b7lxc\" (UID: \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:01.030011 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.029967    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8882\" (UniqueName: \"kubernetes.io/projected/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-kube-api-access-z8882\") pod \"maas-api-revoke-sub-d561d836-b7lxc\" (UID: \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:01.030548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.030525    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d561d836-b7lxc\" (UID: \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:01.032486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.032461    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d561d836-b7lxc\" (UID: \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:01.038348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.038320    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8882\" (UniqueName: \"kubernetes.io/projected/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-kube-api-access-z8882\") pod \"maas-api-revoke-sub-d561d836-b7lxc\" (UID: \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:01.114120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.114034    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:01.407250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.407208    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"]
Oct 30 21:39:01.409060 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:39:01.409032    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99d75b1_b597_4f77_99d1_1b3ef2f799a5.slice/crio-12e49172d99299a28ef0e6ad9e9d42b2ff69bafd700d6757a67cdc3447d96d99 WatchSource:0}: Error finding container 12e49172d99299a28ef0e6ad9e9d42b2ff69bafd700d6757a67cdc3447d96d99: Status 404 returned error can't find the container with id 12e49172d99299a28ef0e6ad9e9d42b2ff69bafd700d6757a67cdc3447d96d99
Oct 30 21:39:01.440406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.440366    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:39:01.482805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.482775    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-openshift-ca\") pod \"03af4727-db4d-40d4-bbc5-8e13561cdf14\" (UID: \"03af4727-db4d-40d4-bbc5-8e13561cdf14\") "
Oct 30 21:39:01.482991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.482842    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-xks-ca\") pod \"03af4727-db4d-40d4-bbc5-8e13561cdf14\" (UID: \"03af4727-db4d-40d4-bbc5-8e13561cdf14\") "
Oct 30 21:39:01.483135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.483111    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-openshift-ca" pod "03af4727-db4d-40d4-bbc5-8e13561cdf14" (UID: "03af4727-db4d-40d4-bbc5-8e13561cdf14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:01.483202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.483130    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03af4727-db4d-40d4-bbc5-8e13561cdf14-kube-api-access-h6wjg\" (UniqueName: \"kubernetes.io/projected/03af4727-db4d-40d4-bbc5-8e13561cdf14-kube-api-access-h6wjg\") pod \"03af4727-db4d-40d4-bbc5-8e13561cdf14\" (UID: \"03af4727-db4d-40d4-bbc5-8e13561cdf14\") "
Oct 30 21:39:01.484194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.484168    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:01.485301 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.485275    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-xks-ca" pod "03af4727-db4d-40d4-bbc5-8e13561cdf14" (UID: "03af4727-db4d-40d4-bbc5-8e13561cdf14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:01.485515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.485496    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03af4727-db4d-40d4-bbc5-8e13561cdf14-kube-api-access-h6wjg" pod "03af4727-db4d-40d4-bbc5-8e13561cdf14" (UID: "03af4727-db4d-40d4-bbc5-8e13561cdf14"). InnerVolumeSpecName "kube-api-access-h6wjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:01.586155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.586118    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6wjg\" (UniqueName: \"kubernetes.io/projected/03af4727-db4d-40d4-bbc5-8e13561cdf14-kube-api-access-h6wjg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:01.586155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:01.586158    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03af4727-db4d-40d4-bbc5-8e13561cdf14-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:02.149893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.149848    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc" event={"ID":"a99d75b1-b597-4f77-99d1-1b3ef2f799a5","Type":"ContainerStarted","Data":"7405659c94bbe938893f778859afb2390e6dbd0352145846cc821b6c57e195a7"}
Oct 30 21:39:02.150313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.149918    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc" event={"ID":"a99d75b1-b597-4f77-99d1-1b3ef2f799a5","Type":"ContainerStarted","Data":"12e49172d99299a28ef0e6ad9e9d42b2ff69bafd700d6757a67cdc3447d96d99"}
Oct 30 21:39:02.151488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.151463    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"
Oct 30 21:39:02.151609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.151462    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4" event={"ID":"03af4727-db4d-40d4-bbc5-8e13561cdf14","Type":"ContainerDied","Data":"398d05f4248095fb196b3c2e8a99edaeb37159634036133fa0f176d474e60473"}
Oct 30 21:39:02.151609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.151592    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="398d05f4248095fb196b3c2e8a99edaeb37159634036133fa0f176d474e60473"
Oct 30 21:39:02.170700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.170657    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc" podStartSLOduration=2.170643252 podStartE2EDuration="2.170643252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:02.167336308 +0000 UTC m=+2255.213895487" watchObservedRunningTime="2026-10-30 21:39:02.170643252 +0000 UTC m=+2255.217202439"
Oct 30 21:39:02.250417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.250171    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2" podUID="924e845c-e3ad-4ac8-9991-24e5fb40b7b8" containerName="payload-processing" containerID="cri-o://6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f" gracePeriod=25
Oct 30 21:39:02.698931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.698908    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:39:02.711627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.711599    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-routing-overlay-volume\") pod \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") "
Oct 30 21:39:02.711914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.711901    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-provider-credentials\") pod \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") "
Oct 30 21:39:02.711964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.711942    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-plugins-config-volume\") pod \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") "
Oct 30 21:39:02.712019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.711900    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-routing-overlay-volume" pod "924e845c-e3ad-4ac8-9991-24e5fb40b7b8" (UID: "924e845c-e3ad-4ac8-9991-24e5fb40b7b8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:02.712019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.711994    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-kube-api-access-p5f9g\" (UniqueName: \"kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-kube-api-access-p5f9g\") pod \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\" (UID: \"924e845c-e3ad-4ac8-9991-24e5fb40b7b8\") "
Oct 30 21:39:02.712374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.712350    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-plugins-config-volume" pod "924e845c-e3ad-4ac8-9991-24e5fb40b7b8" (UID: "924e845c-e3ad-4ac8-9991-24e5fb40b7b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:02.712661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.712633    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:02.712733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.712666    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:02.714021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.714003    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-provider-credentials" pod "924e845c-e3ad-4ac8-9991-24e5fb40b7b8" (UID: "924e845c-e3ad-4ac8-9991-24e5fb40b7b8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:02.714352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.714333    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-kube-api-access-p5f9g" pod "924e845c-e3ad-4ac8-9991-24e5fb40b7b8" (UID: "924e845c-e3ad-4ac8-9991-24e5fb40b7b8"). InnerVolumeSpecName "kube-api-access-p5f9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:02.813878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.813838    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:02.813878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:02.813874    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5f9g\" (UniqueName: \"kubernetes.io/projected/924e845c-e3ad-4ac8-9991-24e5fb40b7b8-kube-api-access-p5f9g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:03.165139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:03.165112    4918 generic.go:356] "Generic (PLEG): container finished" podID="a99d75b1-b597-4f77-99d1-1b3ef2f799a5" containerID="7405659c94bbe938893f778859afb2390e6dbd0352145846cc821b6c57e195a7" exitCode=0
Oct 30 21:39:03.165534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:03.165196    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc" event={"ID":"a99d75b1-b597-4f77-99d1-1b3ef2f799a5","Type":"ContainerDied","Data":"7405659c94bbe938893f778859afb2390e6dbd0352145846cc821b6c57e195a7"}
Oct 30 21:39:03.166654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:03.166632    4918 generic.go:356] "Generic (PLEG): container finished" podID="924e845c-e3ad-4ac8-9991-24e5fb40b7b8" containerID="6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f" exitCode=0
Oct 30 21:39:03.166786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:03.166670    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"
Oct 30 21:39:03.166786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:03.166683    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2" event={"ID":"924e845c-e3ad-4ac8-9991-24e5fb40b7b8","Type":"ContainerDied","Data":"6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f"}
Oct 30 21:39:03.166786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:03.166728    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2" event={"ID":"924e845c-e3ad-4ac8-9991-24e5fb40b7b8","Type":"ContainerDied","Data":"5fe5d15c8e6363fb5d374174bc550595224af74fb29c330b65757dac9b3c61a4"}
Oct 30 21:39:03.166786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:03.166756    4918 scope.go:122] "RemoveContainer" containerID="6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f"
Oct 30 21:39:03.178525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:03.178508    4918 scope.go:122] "RemoveContainer" containerID="6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f"
Oct 30 21:39:03.178827 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:03.178805    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f\": container with ID starting with 6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f not found: ID does not exist" containerID="6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f"
Oct 30 21:39:03.178877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:03.178831    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f"} err="failed to get container status \"6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f\": rpc error: code = NotFound desc = could not find container \"6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f\": container with ID starting with 6971e388d435608c32c559f1f14ebfbc819d25cf1209f7d6b3fb8b5e2cf8d72f not found: ID does not exist"
Oct 30 21:39:03.201535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:03.200609    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"]
Oct 30 21:39:03.206794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:03.206773    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-5588bfft2"]
Oct 30 21:39:03.937874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:03.937847    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:39:04.363486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:04.363462    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:04.449988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:04.449933    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-kube-api-access-z8882\" (UniqueName: \"kubernetes.io/projected/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-kube-api-access-z8882\") pod \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\" (UID: \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\") "
Oct 30 21:39:04.450202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:04.450074    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-openshift-ca\") pod \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\" (UID: \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\") "
Oct 30 21:39:04.450202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:04.450163    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-xks-ca\") pod \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\" (UID: \"a99d75b1-b597-4f77-99d1-1b3ef2f799a5\") "
Oct 30 21:39:04.450481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:04.450456    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-openshift-ca" pod "a99d75b1-b597-4f77-99d1-1b3ef2f799a5" (UID: "a99d75b1-b597-4f77-99d1-1b3ef2f799a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:04.451333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:04.451306    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.452540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:04.452517    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-xks-ca" pod "a99d75b1-b597-4f77-99d1-1b3ef2f799a5" (UID: "a99d75b1-b597-4f77-99d1-1b3ef2f799a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:04.452716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:04.452626    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-kube-api-access-z8882" pod "a99d75b1-b597-4f77-99d1-1b3ef2f799a5" (UID: "a99d75b1-b597-4f77-99d1-1b3ef2f799a5"). InnerVolumeSpecName "kube-api-access-z8882". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:04.554259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:04.554123    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8882\" (UniqueName: \"kubernetes.io/projected/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-kube-api-access-z8882\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.554259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:04.554159    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99d75b1-b597-4f77-99d1-1b3ef2f799a5-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.945678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:04.945594    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:39:05.001682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:05.001653    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"]
Oct 30 21:39:05.081922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:05.081889    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924e845c-e3ad-4ac8-9991-24e5fb40b7b8" path="/var/lib/kubelet/pods/924e845c-e3ad-4ac8-9991-24e5fb40b7b8/volumes"
Oct 30 21:39:05.187364 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:05.187333    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"
Oct 30 21:39:05.187589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:05.187371    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc" event={"ID":"a99d75b1-b597-4f77-99d1-1b3ef2f799a5","Type":"ContainerDied","Data":"12e49172d99299a28ef0e6ad9e9d42b2ff69bafd700d6757a67cdc3447d96d99"}
Oct 30 21:39:05.187589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:05.187445    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12e49172d99299a28ef0e6ad9e9d42b2ff69bafd700d6757a67cdc3447d96d99"
Oct 30 21:39:08.769488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:08.769447    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"]
Oct 30 21:39:08.777184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:08.777154    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"]
Oct 30 21:39:08.780254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:08.780224    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:39:09.174693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:09.174605    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:39:10.002322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.002274    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv" podUID="c4f46782-db4b-4cdd-8bcf-d22613c9e0c1" containerName="payload-processing" containerID="cri-o://d05f72b7792cee96371da6cb63dafad713452cb41bc01ea70d612f58637740ab" gracePeriod=25
Oct 30 21:39:10.026150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.026116    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"]
Oct 30 21:39:10.027029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.027010    4918 cpu_manager.go:414] "containerMap: removing container" podUID="03af4727-db4d-40d4-bbc5-8e13561cdf14" containerName="revoke-keys"
Oct 30 21:39:10.027029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.027028    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03af4727-db4d-40d4-bbc5-8e13561cdf14" containerName="revoke-keys"
Oct 30 21:39:10.027736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.027072    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a99d75b1-b597-4f77-99d1-1b3ef2f799a5" containerName="revoke-keys"
Oct 30 21:39:10.027736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.027077    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99d75b1-b597-4f77-99d1-1b3ef2f799a5" containerName="revoke-keys"
Oct 30 21:39:10.027736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.027101    4918 cpu_manager.go:414] "containerMap: removing container" podUID="924e845c-e3ad-4ac8-9991-24e5fb40b7b8" containerName="payload-processing"
Oct 30 21:39:10.027736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.027106    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924e845c-e3ad-4ac8-9991-24e5fb40b7b8" containerName="payload-processing"
Oct 30 21:39:10.027736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.027237    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99d75b1-b597-4f77-99d1-1b3ef2f799a5" containerName="revoke-keys"
Oct 30 21:39:10.027736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.027254    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="924e845c-e3ad-4ac8-9991-24e5fb40b7b8" containerName="payload-processing"
Oct 30 21:39:10.027736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.027264    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="03af4727-db4d-40d4-bbc5-8e13561cdf14" containerName="revoke-keys"
Oct 30 21:39:10.031983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.031964    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.046203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.046176    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"agc-cred-3ca8bb326130\""
Oct 30 21:39:10.046346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.046301    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a378a2\"/\"payload-processing-external-model-e2e--d1dc7551-dockercfg-hsscw\""
Oct 30 21:39:10.048873 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.048848    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"]
Oct 30 21:39:10.117596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.117561    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.117756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.117683    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchs6\" (UniqueName: \"kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-kube-api-access-fchs6\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.117756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.117722    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.117884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.117852    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.297414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.259372    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.384603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.384547    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchs6\" (UniqueName: \"kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-kube-api-access-fchs6\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.385377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.385349    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.386470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.386449    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.461692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.386294    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.461692 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:10.331549    4918 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a378a2/agc-cred-3ca8bb326130: secret "agc-cred-3ca8bb326130" not found
Oct 30 21:39:10.461692 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:10.446982    4918 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk: secret "agc-cred-3ca8bb326130" not found
Oct 30 21:39:10.461692 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:10.447069    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-provider-credentials podName:8ddbbe50-4203-4539-9215-9a4fd3d8576f nodeName:}" failed. No retries permitted until 2026-10-30 21:39:10.947046012 +0000 UTC m=+2263.993605150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk" (UID: "8ddbbe50-4203-4539-9215-9a4fd3d8576f") : secret "agc-cred-3ca8bb326130" not found
Oct 30 21:39:10.461692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.387304    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.631485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.618695    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchs6\" (UniqueName: \"kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-kube-api-access-fchs6\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.638631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.626795    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv" event={"ID":"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1","Type":"ContainerDied","Data":"d05f72b7792cee96371da6cb63dafad713452cb41bc01ea70d612f58637740ab"}
Oct 30 21:39:10.638631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.626750    4918 generic.go:356] "Generic (PLEG): container finished" podID="c4f46782-db4b-4cdd-8bcf-d22613c9e0c1" containerID="d05f72b7792cee96371da6cb63dafad713452cb41bc01ea70d612f58637740ab" exitCode=0
Oct 30 21:39:10.674032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.673998    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:39:10.701744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.701716    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"]
Oct 30 21:39:10.702662 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:10.702594    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk" podUID="8ddbbe50-4203-4539-9215-9a4fd3d8576f"
Oct 30 21:39:10.765172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.765141    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-provider-credentials\") pod \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") "
Oct 30 21:39:10.765307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.765251    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-plugins-config-volume\") pod \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") "
Oct 30 21:39:10.765307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.765278    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-kube-api-access-n8ch5\" (UniqueName: \"kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-kube-api-access-n8ch5\") pod \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") "
Oct 30 21:39:10.765457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.765441    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-routing-overlay-volume\") pod \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\" (UID: \"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1\") "
Oct 30 21:39:10.765727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.765645    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-plugins-config-volume" pod "c4f46782-db4b-4cdd-8bcf-d22613c9e0c1" (UID: "c4f46782-db4b-4cdd-8bcf-d22613c9e0c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:10.765842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.765794    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-routing-overlay-volume" pod "c4f46782-db4b-4cdd-8bcf-d22613c9e0c1" (UID: "c4f46782-db4b-4cdd-8bcf-d22613c9e0c1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:10.766288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.766266    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:10.766378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.766290    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:10.767709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.767688    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-provider-credentials" pod "c4f46782-db4b-4cdd-8bcf-d22613c9e0c1" (UID: "c4f46782-db4b-4cdd-8bcf-d22613c9e0c1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:10.767818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.767726    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-kube-api-access-n8ch5" pod "c4f46782-db4b-4cdd-8bcf-d22613c9e0c1" (UID: "c4f46782-db4b-4cdd-8bcf-d22613c9e0c1"). InnerVolumeSpecName "kube-api-access-n8ch5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:10.875417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.871947    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:10.875417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.871996    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8ch5\" (UniqueName: \"kubernetes.io/projected/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1-kube-api-access-n8ch5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:10.985419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:10.974906    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:10.985419 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:10.975440    4918 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a378a2/agc-cred-3ca8bb326130: secret "agc-cred-3ca8bb326130" not found
Oct 30 21:39:10.985419 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:10.975460    4918 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk: secret "agc-cred-3ca8bb326130" not found
Oct 30 21:39:10.985419 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:10.975536    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-provider-credentials podName:8ddbbe50-4203-4539-9215-9a4fd3d8576f nodeName:}" failed. No retries permitted until 2026-10-30 21:39:11.975516055 +0000 UTC m=+2265.022075193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk" (UID: "8ddbbe50-4203-4539-9215-9a4fd3d8576f") : secret "agc-cred-3ca8bb326130" not found
Oct 30 21:39:11.645635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.645605    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"
Oct 30 21:39:11.646126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.645640    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv" event={"ID":"c4f46782-db4b-4cdd-8bcf-d22613c9e0c1","Type":"ContainerDied","Data":"23af41e2a4b6762661967af663154890eb3fb956e481d75cd4819f76ca78a24c"}
Oct 30 21:39:11.646126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.645696    4918 scope.go:122] "RemoveContainer" containerID="d05f72b7792cee96371da6cb63dafad713452cb41bc01ea70d612f58637740ab"
Oct 30 21:39:11.646126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.645763    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:11.656913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.656895    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:11.692275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.692244    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"]
Oct 30 21:39:11.699620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.699597    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-7fb7w5gsv"]
Oct 30 21:39:11.716782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.716759    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-routing-overlay-volume\") pod \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") "
Oct 30 21:39:11.716886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.716875    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-kube-api-access-fchs6\" (UniqueName: \"kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-kube-api-access-fchs6\") pod \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") "
Oct 30 21:39:11.717040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.717028    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-plugins-config-volume\") pod \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") "
Oct 30 21:39:11.717175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.717152    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-routing-overlay-volume" pod "8ddbbe50-4203-4539-9215-9a4fd3d8576f" (UID: "8ddbbe50-4203-4539-9215-9a4fd3d8576f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:11.717432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.717377    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-plugins-config-volume" pod "8ddbbe50-4203-4539-9215-9a4fd3d8576f" (UID: "8ddbbe50-4203-4539-9215-9a4fd3d8576f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:11.717737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.717722    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:11.717786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.717741    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ddbbe50-4203-4539-9215-9a4fd3d8576f-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:11.719126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.719105    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-kube-api-access-fchs6" pod "8ddbbe50-4203-4539-9215-9a4fd3d8576f" (UID: "8ddbbe50-4203-4539-9215-9a4fd3d8576f"). InnerVolumeSpecName "kube-api-access-fchs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:11.819344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:11.819305    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fchs6\" (UniqueName: \"kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-kube-api-access-fchs6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:12.023419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:12.023372    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk\" (UID: \"8ddbbe50-4203-4539-9215-9a4fd3d8576f\") " pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:12.023630 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:12.023484    4918 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a378a2/agc-cred-3ca8bb326130: secret "agc-cred-3ca8bb326130" not found
Oct 30 21:39:12.023630 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:12.023501    4918 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk: secret "agc-cred-3ca8bb326130" not found
Oct 30 21:39:12.023630 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:12.023548    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-provider-credentials podName:8ddbbe50-4203-4539-9215-9a4fd3d8576f nodeName:}" failed. No retries permitted until 2026-10-30 21:39:14.0235352 +0000 UTC m=+2267.070094323 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk" (UID: "8ddbbe50-4203-4539-9215-9a4fd3d8576f") : secret "agc-cred-3ca8bb326130" not found
Oct 30 21:39:12.727716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:12.727687    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"
Oct 30 21:39:12.956488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:12.956455    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"]
Oct 30 21:39:13.053468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:13.049606    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7f7bpdrnk"]
Oct 30 21:39:13.103421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:13.102727    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ddbbe50-4203-4539-9215-9a4fd3d8576f" path="/var/lib/kubelet/pods/8ddbbe50-4203-4539-9215-9a4fd3d8576f/volumes"
Oct 30 21:39:13.103421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:13.103232    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f46782-db4b-4cdd-8bcf-d22613c9e0c1" path="/var/lib/kubelet/pods/c4f46782-db4b-4cdd-8bcf-d22613c9e0c1/volumes"
Oct 30 21:39:13.203598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:13.203570    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ddbbe50-4203-4539-9215-9a4fd3d8576f-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:13.819424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:13.790864    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls" podUID="df3d0b3d-b641-42a0-81af-2643ae1d1b3c" containerName="payload-processing" containerID="cri-o://3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230" gracePeriod=25
Oct 30 21:39:13.819424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:13.791128    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm" podUID="25b1dc8f-ad62-4490-b18d-0213bd1f137e" containerName="payload-processing" containerID="cri-o://05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13" gracePeriod=25
Oct 30 21:39:14.505640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.505614    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:39:14.510210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.510185    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:39:14.616253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.616212    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"]
Oct 30 21:39:14.617163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.617137    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c4f46782-db4b-4cdd-8bcf-d22613c9e0c1" containerName="payload-processing"
Oct 30 21:39:14.617163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.617163    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f46782-db4b-4cdd-8bcf-d22613c9e0c1" containerName="payload-processing"
Oct 30 21:39:14.617337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.617236    4918 cpu_manager.go:414] "containerMap: removing container" podUID="25b1dc8f-ad62-4490-b18d-0213bd1f137e" containerName="payload-processing"
Oct 30 21:39:14.617337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.617244    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b1dc8f-ad62-4490-b18d-0213bd1f137e" containerName="payload-processing"
Oct 30 21:39:14.617337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.617286    4918 cpu_manager.go:414] "containerMap: removing container" podUID="df3d0b3d-b641-42a0-81af-2643ae1d1b3c" containerName="payload-processing"
Oct 30 21:39:14.617337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.617294    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df3d0b3d-b641-42a0-81af-2643ae1d1b3c" containerName="payload-processing"
Oct 30 21:39:14.617573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.617522    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="df3d0b3d-b641-42a0-81af-2643ae1d1b3c" containerName="payload-processing"
Oct 30 21:39:14.617573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.617540    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f46782-db4b-4cdd-8bcf-d22613c9e0c1" containerName="payload-processing"
Oct 30 21:39:14.617573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.617565    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b1dc8f-ad62-4490-b18d-0213bd1f137e" containerName="payload-processing"
Oct 30 21:39:14.621615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.621596    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:14.630291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.630206    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:39:14.630291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.630212    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:39:14.630589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.630564    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:39:14.630754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.630626    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:39:14.630754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.630635    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fa4045c246e1\""
Oct 30 21:39:14.630754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.630697    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-jgk7c\""
Oct 30 21:39:14.638443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.637767    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"]
Oct 30 21:39:14.666882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.666855    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-routing-overlay-volume\") pod \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") "
Oct 30 21:39:14.667069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.667050    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-kube-api-access-bxcdd\" (UniqueName: \"kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-kube-api-access-bxcdd\") pod \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") "
Oct 30 21:39:14.667222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.667110    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-provider-credentials\") pod \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") "
Oct 30 21:39:14.667222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.667167    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-plugins-config-volume\") pod \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\" (UID: \"25b1dc8f-ad62-4490-b18d-0213bd1f137e\") "
Oct 30 21:39:14.667222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.667204    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-routing-overlay-volume\") pod \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") "
Oct 30 21:39:14.667441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.667227    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-routing-overlay-volume" pod "25b1dc8f-ad62-4490-b18d-0213bd1f137e" (UID: "25b1dc8f-ad62-4490-b18d-0213bd1f137e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:14.667534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.667520    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-provider-credentials\") pod \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") "
Oct 30 21:39:14.667760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.667733    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-plugins-config-volume" pod "25b1dc8f-ad62-4490-b18d-0213bd1f137e" (UID: "25b1dc8f-ad62-4490-b18d-0213bd1f137e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:14.667872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.667848    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-routing-overlay-volume" pod "df3d0b3d-b641-42a0-81af-2643ae1d1b3c" (UID: "df3d0b3d-b641-42a0-81af-2643ae1d1b3c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:14.667946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.667883    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-plugins-config-volume\") pod \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") "
Oct 30 21:39:14.668000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.667942    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-kube-api-access-6qp88\" (UniqueName: \"kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-kube-api-access-6qp88\") pod \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\" (UID: \"df3d0b3d-b641-42a0-81af-2643ae1d1b3c\") "
Oct 30 21:39:14.668485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.668458    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-plugins-config-volume" pod "df3d0b3d-b641-42a0-81af-2643ae1d1b3c" (UID: "df3d0b3d-b641-42a0-81af-2643ae1d1b3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:14.669105 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.669080    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:14.669218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.669112    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:14.669218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.669128    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:14.669218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.669142    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25b1dc8f-ad62-4490-b18d-0213bd1f137e-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:14.670115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.670089    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-kube-api-access-bxcdd" pod "25b1dc8f-ad62-4490-b18d-0213bd1f137e" (UID: "25b1dc8f-ad62-4490-b18d-0213bd1f137e"). InnerVolumeSpecName "kube-api-access-bxcdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:14.670204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.670134    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-provider-credentials" pod "25b1dc8f-ad62-4490-b18d-0213bd1f137e" (UID: "25b1dc8f-ad62-4490-b18d-0213bd1f137e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:14.670252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.670201    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-provider-credentials" pod "df3d0b3d-b641-42a0-81af-2643ae1d1b3c" (UID: "df3d0b3d-b641-42a0-81af-2643ae1d1b3c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:14.670777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.670758    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-kube-api-access-6qp88" pod "df3d0b3d-b641-42a0-81af-2643ae1d1b3c" (UID: "df3d0b3d-b641-42a0-81af-2643ae1d1b3c"). InnerVolumeSpecName "kube-api-access-6qp88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:14.772069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.771726    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-plugins-config-volume\") pod \"payload-processing-external-model-68c55b6465-6g2zw\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") " pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:14.772069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.771892    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-routing-overlay-volume\") pod \"payload-processing-external-model-68c55b6465-6g2zw\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") " pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:14.772069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.771942    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-provider-credentials\") pod \"payload-processing-external-model-68c55b6465-6g2zw\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") " pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:14.772069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.771990    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbwsm\" (UniqueName: \"kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-kube-api-access-hbwsm\") pod \"payload-processing-external-model-68c55b6465-6g2zw\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") " pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:14.795128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.794784    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:14.795128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.794825    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qp88\" (UniqueName: \"kubernetes.io/projected/df3d0b3d-b641-42a0-81af-2643ae1d1b3c-kube-api-access-6qp88\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:14.795128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.794842    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxcdd\" (UniqueName: \"kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-kube-api-access-bxcdd\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:14.795128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.794857    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25b1dc8f-ad62-4490-b18d-0213bd1f137e-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:14.901427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.899466    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-plugins-config-volume\") pod \"payload-processing-external-model-68c55b6465-6g2zw\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") " pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:14.901427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.899567    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-routing-overlay-volume\") pod \"payload-processing-external-model-68c55b6465-6g2zw\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") " pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:14.901427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.899603    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-provider-credentials\") pod \"payload-processing-external-model-68c55b6465-6g2zw\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") " pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:14.901427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.899653    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbwsm\" (UniqueName: \"kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-kube-api-access-hbwsm\") pod \"payload-processing-external-model-68c55b6465-6g2zw\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") " pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:14.901427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.900438    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-plugins-config-volume\") pod \"payload-processing-external-model-68c55b6465-6g2zw\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") " pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:14.901427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.900784    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-routing-overlay-volume\") pod \"payload-processing-external-model-68c55b6465-6g2zw\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") " pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:14.951415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:14.938077    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbwsm\" (UniqueName: \"kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-kube-api-access-hbwsm\") pod \"payload-processing-external-model-68c55b6465-6g2zw\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") " pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:15.015411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.009994    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-provider-credentials\") pod \"payload-processing-external-model-68c55b6465-6g2zw\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") " pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:15.218467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.218076    4918 generic.go:356] "Generic (PLEG): container finished" podID="df3d0b3d-b641-42a0-81af-2643ae1d1b3c" containerID="3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230" exitCode=0
Oct 30 21:39:15.218467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.218354    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls" event={"ID":"df3d0b3d-b641-42a0-81af-2643ae1d1b3c","Type":"ContainerDied","Data":"3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230"}
Oct 30 21:39:15.218716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.218482    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls" event={"ID":"df3d0b3d-b641-42a0-81af-2643ae1d1b3c","Type":"ContainerDied","Data":"5c68495ad2fca4b2929aab8e6f806eb0d3069f1f6babdd9a84df65a727116f8a"}
Oct 30 21:39:15.218716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.218537    4918 scope.go:122] "RemoveContainer" containerID="3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230"
Oct 30 21:39:15.218716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.218539    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"
Oct 30 21:39:15.223420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.223371    4918 generic.go:356] "Generic (PLEG): container finished" podID="25b1dc8f-ad62-4490-b18d-0213bd1f137e" containerID="05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13" exitCode=0
Oct 30 21:39:15.224016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.223798    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm" event={"ID":"25b1dc8f-ad62-4490-b18d-0213bd1f137e","Type":"ContainerDied","Data":"05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13"}
Oct 30 21:39:15.224228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.224200    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm" event={"ID":"25b1dc8f-ad62-4490-b18d-0213bd1f137e","Type":"ContainerDied","Data":"2708cebfbc927ab53c2f3b2c37cdd0366b816d33e7d66ee9ca509a4bcf5cd3b1"}
Oct 30 21:39:15.226138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.225608    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"
Oct 30 21:39:15.240042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.239794    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:15.277130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.273142    4918 scope.go:122] "RemoveContainer" containerID="3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230"
Oct 30 21:39:15.277130 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:15.275441    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230\": container with ID starting with 3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230 not found: ID does not exist" containerID="3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230"
Oct 30 21:39:15.277130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.275482    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230"} err="failed to get container status \"3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230\": rpc error: code = NotFound desc = could not find container \"3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230\": container with ID starting with 3a53d43706f2282cc4c9ea731b2e27966c06250fb79845cfe91f63fa8fd78230 not found: ID does not exist"
Oct 30 21:39:15.277130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.275505    4918 scope.go:122] "RemoveContainer" containerID="05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13"
Oct 30 21:39:15.283245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.283218    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"]
Oct 30 21:39:15.294205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.294141    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-6cdbsqwls"]
Oct 30 21:39:15.300238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.300214    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"]
Oct 30 21:39:15.304202 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:15.302984    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b1dc8f_ad62_4490_b18d_0213bd1f137e.slice/crio-2708cebfbc927ab53c2f3b2c37cdd0366b816d33e7d66ee9ca509a4bcf5cd3b1\": RecentStats: unable to find data in memory cache]"
Oct 30 21:39:15.305132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.305109    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a378a2/payload-processing-external-model-e2e-extmod-a-a378a2-7977wzvkm"]
Oct 30 21:39:15.312829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.312808    4918 scope.go:122] "RemoveContainer" containerID="05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13"
Oct 30 21:39:15.313140 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:15.313120    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13\": container with ID starting with 05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13 not found: ID does not exist" containerID="05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13"
Oct 30 21:39:15.313222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.313147    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13"} err="failed to get container status \"05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13\": rpc error: code = NotFound desc = could not find container \"05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13\": container with ID starting with 05a9837a2d9096fa28c8708a6ae829fbc4a3d898d8bd4829dcfbf2c104035d13 not found: ID does not exist"
Oct 30 21:39:15.558206 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:39:15.558172    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b142a0f_0389_433b_8260_304cf65ea08a.slice/crio-682a0f0c09ebdf88b6b71d1f8d4d8672c1a34dbe6f09dd71a5fed9bc26658916 WatchSource:0}: Error finding container 682a0f0c09ebdf88b6b71d1f8d4d8672c1a34dbe6f09dd71a5fed9bc26658916: Status 404 returned error can't find the container with id 682a0f0c09ebdf88b6b71d1f8d4d8672c1a34dbe6f09dd71a5fed9bc26658916
Oct 30 21:39:15.559438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:15.559412    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"]
Oct 30 21:39:16.323415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:16.321140    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw" event={"ID":"7b142a0f-0389-433b-8260-304cf65ea08a","Type":"ContainerStarted","Data":"e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5"}
Oct 30 21:39:16.323415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:16.321217    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw" event={"ID":"7b142a0f-0389-433b-8260-304cf65ea08a","Type":"ContainerStarted","Data":"682a0f0c09ebdf88b6b71d1f8d4d8672c1a34dbe6f09dd71a5fed9bc26658916"}
Oct 30 21:39:16.323415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:16.322670    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:16.366696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:16.366635    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw" podStartSLOduration=2.366615687 podStartE2EDuration="2.366615687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:16.351041961 +0000 UTC m=+2269.397601120" watchObservedRunningTime="2026-10-30 21:39:16.366615687 +0000 UTC m=+2269.413174833"
Oct 30 21:39:17.174025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:17.173971    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b1dc8f-ad62-4490-b18d-0213bd1f137e" path="/var/lib/kubelet/pods/25b1dc8f-ad62-4490-b18d-0213bd1f137e/volumes"
Oct 30 21:39:17.174863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:17.174838    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df3d0b3d-b641-42a0-81af-2643ae1d1b3c" path="/var/lib/kubelet/pods/df3d0b3d-b641-42a0-81af-2643ae1d1b3c/volumes"
Oct 30 21:39:23.438423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.435645    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"]
Oct 30 21:39:23.443781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.443753    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:23.466096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.450313    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"]
Oct 30 21:39:23.466096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.456340    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:39:23.466096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.459291    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:23.466096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.459378    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:23.466096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.459451    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:39:23.507200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.507173    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc417a70-jszc5\" (UID: \"b6104801-7f62-47d0-ab21-efaf9cda4484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:23.507379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.507308    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc417a70-jszc5\" (UID: \"b6104801-7f62-47d0-ab21-efaf9cda4484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:23.507845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.507815    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkn6r\" (UniqueName: \"kubernetes.io/projected/b6104801-7f62-47d0-ab21-efaf9cda4484-kube-api-access-kkn6r\") pod \"maas-api-revoke-sub-fc417a70-jszc5\" (UID: \"b6104801-7f62-47d0-ab21-efaf9cda4484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:23.609885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.609854    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkn6r\" (UniqueName: \"kubernetes.io/projected/b6104801-7f62-47d0-ab21-efaf9cda4484-kube-api-access-kkn6r\") pod \"maas-api-revoke-sub-fc417a70-jszc5\" (UID: \"b6104801-7f62-47d0-ab21-efaf9cda4484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:23.610101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.609905    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc417a70-jszc5\" (UID: \"b6104801-7f62-47d0-ab21-efaf9cda4484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:23.610101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.609957    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc417a70-jszc5\" (UID: \"b6104801-7f62-47d0-ab21-efaf9cda4484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:23.610651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.610625    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc417a70-jszc5\" (UID: \"b6104801-7f62-47d0-ab21-efaf9cda4484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:23.612520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.612495    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc417a70-jszc5\" (UID: \"b6104801-7f62-47d0-ab21-efaf9cda4484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:23.619379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.619360    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkn6r\" (UniqueName: \"kubernetes.io/projected/b6104801-7f62-47d0-ab21-efaf9cda4484-kube-api-access-kkn6r\") pod \"maas-api-revoke-sub-fc417a70-jszc5\" (UID: \"b6104801-7f62-47d0-ab21-efaf9cda4484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:23.794990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:23.794963    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:24.100841 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:39:24.100811    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6104801_7f62_47d0_ab21_efaf9cda4484.slice/crio-4fa88bda9baa40da283bd4d43e9ddb525ed440e909c3dc8369d9d38286560ac6 WatchSource:0}: Error finding container 4fa88bda9baa40da283bd4d43e9ddb525ed440e909c3dc8369d9d38286560ac6: Status 404 returned error can't find the container with id 4fa88bda9baa40da283bd4d43e9ddb525ed440e909c3dc8369d9d38286560ac6
Oct 30 21:39:24.102797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:24.102772    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"]
Oct 30 21:39:24.497127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:24.497100    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5" event={"ID":"b6104801-7f62-47d0-ab21-efaf9cda4484","Type":"ContainerStarted","Data":"4fa88bda9baa40da283bd4d43e9ddb525ed440e909c3dc8369d9d38286560ac6"}
Oct 30 21:39:25.567423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:25.565719    4918 generic.go:356] "Generic (PLEG): container finished" podID="b6104801-7f62-47d0-ab21-efaf9cda4484" containerID="cd9e3162c1d8492b997b0f7d616f05ba518f01bc477ba521b911559cd83f5aca" exitCode=0
Oct 30 21:39:25.567423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:25.565940    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5" event={"ID":"b6104801-7f62-47d0-ab21-efaf9cda4484","Type":"ContainerDied","Data":"cd9e3162c1d8492b997b0f7d616f05ba518f01bc477ba521b911559cd83f5aca"}
Oct 30 21:39:26.763833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:26.763809    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:26.880539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:26.880457    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-openshift-ca\") pod \"b6104801-7f62-47d0-ab21-efaf9cda4484\" (UID: \"b6104801-7f62-47d0-ab21-efaf9cda4484\") "
Oct 30 21:39:26.880539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:26.880523    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6104801-7f62-47d0-ab21-efaf9cda4484-kube-api-access-kkn6r\" (UniqueName: \"kubernetes.io/projected/b6104801-7f62-47d0-ab21-efaf9cda4484-kube-api-access-kkn6r\") pod \"b6104801-7f62-47d0-ab21-efaf9cda4484\" (UID: \"b6104801-7f62-47d0-ab21-efaf9cda4484\") "
Oct 30 21:39:26.880732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:26.880710    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-xks-ca\") pod \"b6104801-7f62-47d0-ab21-efaf9cda4484\" (UID: \"b6104801-7f62-47d0-ab21-efaf9cda4484\") "
Oct 30 21:39:26.880826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:26.880805    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-openshift-ca" pod "b6104801-7f62-47d0-ab21-efaf9cda4484" (UID: "b6104801-7f62-47d0-ab21-efaf9cda4484"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:26.881565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:26.881454    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:26.883045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:26.883021    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-xks-ca" pod "b6104801-7f62-47d0-ab21-efaf9cda4484" (UID: "b6104801-7f62-47d0-ab21-efaf9cda4484"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:26.883045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:26.883034    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6104801-7f62-47d0-ab21-efaf9cda4484-kube-api-access-kkn6r" pod "b6104801-7f62-47d0-ab21-efaf9cda4484" (UID: "b6104801-7f62-47d0-ab21-efaf9cda4484"). InnerVolumeSpecName "kube-api-access-kkn6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:26.983464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:26.983437    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkn6r\" (UniqueName: \"kubernetes.io/projected/b6104801-7f62-47d0-ab21-efaf9cda4484-kube-api-access-kkn6r\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:26.983464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:26.983462    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6104801-7f62-47d0-ab21-efaf9cda4484-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:27.584938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:27.584902    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"
Oct 30 21:39:27.584938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:27.584926    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5" event={"ID":"b6104801-7f62-47d0-ab21-efaf9cda4484","Type":"ContainerDied","Data":"4fa88bda9baa40da283bd4d43e9ddb525ed440e909c3dc8369d9d38286560ac6"}
Oct 30 21:39:27.585249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:27.585002    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fa88bda9baa40da283bd4d43e9ddb525ed440e909c3dc8369d9d38286560ac6"
Oct 30 21:39:28.420824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:28.420786    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"]
Oct 30 21:39:28.431921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:28.431879    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:29.167795 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:29.167753    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:39:30.189418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:30.187752    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:30.531427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:30.530516    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:30.650421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:30.643532    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded"
Oct 30 21:39:30.676541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:30.670210    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:39:30.857478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:30.855494    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:30.892543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:30.892490    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:30.919840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:30.919790    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:30.926270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:30.925457    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:30.954501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:30.953478    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:30.962468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:30.961480    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:30.962756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:30.962728    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:39:31.014593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:31.014550    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:31.049905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:31.048913    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:31.071020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:31.070981    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:31.105273 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:31.104984    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 30 21:39:32.145583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:32.145550    4918 scope.go:122] "RemoveContainer" containerID="3cc91604cd09d842359ea900e2e953f510bdaa1dfa42fede700bed538b25994a"
Oct 30 21:39:33.421621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.421563    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l" podUID="9daca20b-16ec-4f06-bcb6-ed98ec77a40b" containerName="payload-processing" containerID="cri-o://5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca" gracePeriod=25
Oct 30 21:39:33.736243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.736218    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:39:33.780091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.780061    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-provider-credentials\") pod \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") "
Oct 30 21:39:33.780243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.780213    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-kube-api-access-7pv9g\" (UniqueName: \"kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-kube-api-access-7pv9g\") pod \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") "
Oct 30 21:39:33.780342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.780325    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-routing-overlay-volume\") pod \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") "
Oct 30 21:39:33.780505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.780481    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-plugins-config-volume\") pod \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\" (UID: \"9daca20b-16ec-4f06-bcb6-ed98ec77a40b\") "
Oct 30 21:39:33.780877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.780851    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-routing-overlay-volume" pod "9daca20b-16ec-4f06-bcb6-ed98ec77a40b" (UID: "9daca20b-16ec-4f06-bcb6-ed98ec77a40b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:33.781256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.781234    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-plugins-config-volume" pod "9daca20b-16ec-4f06-bcb6-ed98ec77a40b" (UID: "9daca20b-16ec-4f06-bcb6-ed98ec77a40b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:33.781451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.781423    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:33.781451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.781441    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:33.782597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.782570    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-provider-credentials" pod "9daca20b-16ec-4f06-bcb6-ed98ec77a40b" (UID: "9daca20b-16ec-4f06-bcb6-ed98ec77a40b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:33.783218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.783195    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-kube-api-access-7pv9g" pod "9daca20b-16ec-4f06-bcb6-ed98ec77a40b" (UID: "9daca20b-16ec-4f06-bcb6-ed98ec77a40b"). InnerVolumeSpecName "kube-api-access-7pv9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:33.974755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.959325    4918 generic.go:356] "Generic (PLEG): container finished" podID="9daca20b-16ec-4f06-bcb6-ed98ec77a40b" containerID="5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca" exitCode=0
Oct 30 21:39:33.974755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.959529    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l" event={"ID":"9daca20b-16ec-4f06-bcb6-ed98ec77a40b","Type":"ContainerDied","Data":"5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca"}
Oct 30 21:39:33.974755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.959583    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l" event={"ID":"9daca20b-16ec-4f06-bcb6-ed98ec77a40b","Type":"ContainerDied","Data":"e0875c99af0dc85159436b05236abb86a8b509a49e0be9c26d86915c461c6bac"}
Oct 30 21:39:33.974755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.959611    4918 scope.go:122] "RemoveContainer" containerID="5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca"
Oct 30 21:39:33.974755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.959785    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"
Oct 30 21:39:33.985259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.980069    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:33.985259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.980099    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pv9g\" (UniqueName: \"kubernetes.io/projected/9daca20b-16ec-4f06-bcb6-ed98ec77a40b-kube-api-access-7pv9g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:33.994135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.994112    4918 scope.go:122] "RemoveContainer" containerID="5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca"
Oct 30 21:39:33.994512 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:33.994481    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca\": container with ID starting with 5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca not found: ID does not exist" containerID="5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca"
Oct 30 21:39:33.994643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:33.994522    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca"} err="failed to get container status \"5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca\": rpc error: code = NotFound desc = could not find container \"5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca\": container with ID starting with 5c402386d84965d927bf03607bb71288356fc045ab643765f86e9d7d03231dca not found: ID does not exist"
Oct 30 21:39:34.014477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.014453    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"]
Oct 30 21:39:34.019398 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.019363    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-36a099/payload-processing-external-model-e2e-extmod-b-36a099-6689p9f5l"]
Oct 30 21:39:34.061294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.061267    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"]
Oct 30 21:39:34.062356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.062335    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b6104801-7f62-47d0-ab21-efaf9cda4484" containerName="revoke-keys"
Oct 30 21:39:34.062356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.062355    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6104801-7f62-47d0-ab21-efaf9cda4484" containerName="revoke-keys"
Oct 30 21:39:34.062580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.062371    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9daca20b-16ec-4f06-bcb6-ed98ec77a40b" containerName="payload-processing"
Oct 30 21:39:34.062580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.062377    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9daca20b-16ec-4f06-bcb6-ed98ec77a40b" containerName="payload-processing"
Oct 30 21:39:34.062580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.062566    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6104801-7f62-47d0-ab21-efaf9cda4484" containerName="revoke-keys"
Oct 30 21:39:34.062580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.062581    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9daca20b-16ec-4f06-bcb6-ed98ec77a40b" containerName="payload-processing"
Oct 30 21:39:34.068624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.068602    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:34.073728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.073702    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:34.073855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.073774    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:34.073855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.073819    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:39:34.074885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.074853    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"]
Oct 30 21:39:34.076617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.076354    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:39:34.182998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.182964    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwm5p\" (UniqueName: \"kubernetes.io/projected/51190d58-33e9-49de-b6ce-4086f1e48254-kube-api-access-qwm5p\") pod \"maas-api-revoke-sub-59086576-j2btd\" (UID: \"51190d58-33e9-49de-b6ce-4086f1e48254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:34.183205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.183185    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59086576-j2btd\" (UID: \"51190d58-33e9-49de-b6ce-4086f1e48254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:34.183358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.183344    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59086576-j2btd\" (UID: \"51190d58-33e9-49de-b6ce-4086f1e48254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:34.285246 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.285158    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59086576-j2btd\" (UID: \"51190d58-33e9-49de-b6ce-4086f1e48254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:34.285484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.285353    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59086576-j2btd\" (UID: \"51190d58-33e9-49de-b6ce-4086f1e48254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:34.285583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.285497    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwm5p\" (UniqueName: \"kubernetes.io/projected/51190d58-33e9-49de-b6ce-4086f1e48254-kube-api-access-qwm5p\") pod \"maas-api-revoke-sub-59086576-j2btd\" (UID: \"51190d58-33e9-49de-b6ce-4086f1e48254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:34.285913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.285893    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59086576-j2btd\" (UID: \"51190d58-33e9-49de-b6ce-4086f1e48254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:34.287930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.287910    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59086576-j2btd\" (UID: \"51190d58-33e9-49de-b6ce-4086f1e48254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:34.293729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.293711    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwm5p\" (UniqueName: \"kubernetes.io/projected/51190d58-33e9-49de-b6ce-4086f1e48254-kube-api-access-qwm5p\") pod \"maas-api-revoke-sub-59086576-j2btd\" (UID: \"51190d58-33e9-49de-b6ce-4086f1e48254\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:34.389180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.389149    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:34.676080 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:39:34.676049    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51190d58_33e9_49de_b6ce_4086f1e48254.slice/crio-3d746f7ab26379e102725030c10d6084b2fbc1b06d17914fe298f498ca918d5b WatchSource:0}: Error finding container 3d746f7ab26379e102725030c10d6084b2fbc1b06d17914fe298f498ca918d5b: Status 404 returned error can't find the container with id 3d746f7ab26379e102725030c10d6084b2fbc1b06d17914fe298f498ca918d5b
Oct 30 21:39:34.678076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.678050    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"]
Oct 30 21:39:34.970306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:34.970225    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd" event={"ID":"51190d58-33e9-49de-b6ce-4086f1e48254","Type":"ContainerStarted","Data":"3d746f7ab26379e102725030c10d6084b2fbc1b06d17914fe298f498ca918d5b"}
Oct 30 21:39:35.081475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.081447    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9daca20b-16ec-4f06-bcb6-ed98ec77a40b" path="/var/lib/kubelet/pods/9daca20b-16ec-4f06-bcb6-ed98ec77a40b/volumes"
Oct 30 21:39:35.216939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.216909    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"]
Oct 30 21:39:35.222047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.222025    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:35.231748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.231724    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"]
Oct 30 21:39:35.308655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.308625    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqkfh\" (UniqueName: \"kubernetes.io/projected/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-kube-api-access-cqkfh\") pod \"maas-api-revoke-sub-1db39592-pwtz9\" (UID: \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:35.308831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.308806    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1db39592-pwtz9\" (UID: \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:35.309002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.308981    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1db39592-pwtz9\" (UID: \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:35.410294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.410258    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1db39592-pwtz9\" (UID: \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:35.410472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.410435    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqkfh\" (UniqueName: \"kubernetes.io/projected/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-kube-api-access-cqkfh\") pod \"maas-api-revoke-sub-1db39592-pwtz9\" (UID: \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:35.410670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.410652    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1db39592-pwtz9\" (UID: \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:35.411293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.411270    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1db39592-pwtz9\" (UID: \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:35.412880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.412861    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1db39592-pwtz9\" (UID: \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:35.420553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.420529    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqkfh\" (UniqueName: \"kubernetes.io/projected/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-kube-api-access-cqkfh\") pod \"maas-api-revoke-sub-1db39592-pwtz9\" (UID: \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:35.542632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.542556    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:35.839215 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:39:35.839186    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6def2c6d_d2fa_4f90_8dcc_4e2a2507e956.slice/crio-4f7d3f99c3d80ef83d52d5afb1f8e9673d93f5407bc3170deb916c8e4294a41c WatchSource:0}: Error finding container 4f7d3f99c3d80ef83d52d5afb1f8e9673d93f5407bc3170deb916c8e4294a41c: Status 404 returned error can't find the container with id 4f7d3f99c3d80ef83d52d5afb1f8e9673d93f5407bc3170deb916c8e4294a41c
Oct 30 21:39:35.840046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.840016    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"]
Oct 30 21:39:35.982858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.982821    4918 generic.go:356] "Generic (PLEG): container finished" podID="51190d58-33e9-49de-b6ce-4086f1e48254" containerID="3455d3e8446247d3ab9cf337bbe4fac349962f5e3ab84cd30636a95c1eb10cd3" exitCode=0
Oct 30 21:39:35.983021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.982905    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd" event={"ID":"51190d58-33e9-49de-b6ce-4086f1e48254","Type":"ContainerDied","Data":"3455d3e8446247d3ab9cf337bbe4fac349962f5e3ab84cd30636a95c1eb10cd3"}
Oct 30 21:39:35.984699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:35.984675    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9" event={"ID":"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956","Type":"ContainerStarted","Data":"4f7d3f99c3d80ef83d52d5afb1f8e9673d93f5407bc3170deb916c8e4294a41c"}
Oct 30 21:39:37.008406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:36.996315    4918 generic.go:356] "Generic (PLEG): container finished" podID="6def2c6d-d2fa-4f90-8dcc-4e2a2507e956" containerID="6375f2777bf6f1f83db960680d06ef63e4236281726338cefe7deb4e9a133455" exitCode=0
Oct 30 21:39:37.008406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:36.997279    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9" event={"ID":"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956","Type":"ContainerDied","Data":"6375f2777bf6f1f83db960680d06ef63e4236281726338cefe7deb4e9a133455"}
Oct 30 21:39:37.254831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:37.254805    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:37.367441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:37.367375    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-openshift-ca\") pod \"51190d58-33e9-49de-b6ce-4086f1e48254\" (UID: \"51190d58-33e9-49de-b6ce-4086f1e48254\") "
Oct 30 21:39:37.367652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:37.367469    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51190d58-33e9-49de-b6ce-4086f1e48254-kube-api-access-qwm5p\" (UniqueName: \"kubernetes.io/projected/51190d58-33e9-49de-b6ce-4086f1e48254-kube-api-access-qwm5p\") pod \"51190d58-33e9-49de-b6ce-4086f1e48254\" (UID: \"51190d58-33e9-49de-b6ce-4086f1e48254\") "
Oct 30 21:39:37.367652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:37.367631    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-xks-ca\") pod \"51190d58-33e9-49de-b6ce-4086f1e48254\" (UID: \"51190d58-33e9-49de-b6ce-4086f1e48254\") "
Oct 30 21:39:37.367805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:37.367778    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-openshift-ca" pod "51190d58-33e9-49de-b6ce-4086f1e48254" (UID: "51190d58-33e9-49de-b6ce-4086f1e48254"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:37.368471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:37.368440    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:37.369774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:37.369754    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-xks-ca" pod "51190d58-33e9-49de-b6ce-4086f1e48254" (UID: "51190d58-33e9-49de-b6ce-4086f1e48254"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:37.369856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:37.369817    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51190d58-33e9-49de-b6ce-4086f1e48254-kube-api-access-qwm5p" pod "51190d58-33e9-49de-b6ce-4086f1e48254" (UID: "51190d58-33e9-49de-b6ce-4086f1e48254"). InnerVolumeSpecName "kube-api-access-qwm5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:37.469856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:37.469769    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51190d58-33e9-49de-b6ce-4086f1e48254-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:37.469856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:37.469799    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwm5p\" (UniqueName: \"kubernetes.io/projected/51190d58-33e9-49de-b6ce-4086f1e48254-kube-api-access-qwm5p\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:38.005941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.005907    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"
Oct 30 21:39:38.006144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.005959    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd" event={"ID":"51190d58-33e9-49de-b6ce-4086f1e48254","Type":"ContainerDied","Data":"3d746f7ab26379e102725030c10d6084b2fbc1b06d17914fe298f498ca918d5b"}
Oct 30 21:39:38.006144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.006028    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d746f7ab26379e102725030c10d6084b2fbc1b06d17914fe298f498ca918d5b"
Oct 30 21:39:38.199763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.199734    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:38.283587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.283515    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-openshift-ca\") pod \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\" (UID: \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\") "
Oct 30 21:39:38.283743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.283605    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-kube-api-access-cqkfh\" (UniqueName: \"kubernetes.io/projected/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-kube-api-access-cqkfh\") pod \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\" (UID: \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\") "
Oct 30 21:39:38.283743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.283721    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-xks-ca\") pod \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\" (UID: \"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956\") "
Oct 30 21:39:38.283997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.283965    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-openshift-ca" pod "6def2c6d-d2fa-4f90-8dcc-4e2a2507e956" (UID: "6def2c6d-d2fa-4f90-8dcc-4e2a2507e956"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:38.284414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.284373    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:38.286100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.286080    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-kube-api-access-cqkfh" pod "6def2c6d-d2fa-4f90-8dcc-4e2a2507e956" (UID: "6def2c6d-d2fa-4f90-8dcc-4e2a2507e956"). InnerVolumeSpecName "kube-api-access-cqkfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:38.286228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.286215    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-xks-ca" pod "6def2c6d-d2fa-4f90-8dcc-4e2a2507e956" (UID: "6def2c6d-d2fa-4f90-8dcc-4e2a2507e956"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:38.386056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.386014    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqkfh\" (UniqueName: \"kubernetes.io/projected/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-kube-api-access-cqkfh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:38.386056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.386051    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:38.618737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:38.611992    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"]
Oct 30 21:39:39.016627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:39.016603    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"
Oct 30 21:39:39.016825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:39.016596    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9" event={"ID":"6def2c6d-d2fa-4f90-8dcc-4e2a2507e956","Type":"ContainerDied","Data":"4f7d3f99c3d80ef83d52d5afb1f8e9673d93f5407bc3170deb916c8e4294a41c"}
Oct 30 21:39:39.016825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:39.016728    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f7d3f99c3d80ef83d52d5afb1f8e9673d93f5407bc3170deb916c8e4294a41c"
Oct 30 21:39:43.612323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:43.612284    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw" podUID="7b142a0f-0389-433b-8260-304cf65ea08a" containerName="payload-processing" containerID="cri-o://e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5" gracePeriod=25
Oct 30 21:39:44.005257 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.005235    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:44.025313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.025286    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-routing-overlay-volume\") pod \"7b142a0f-0389-433b-8260-304cf65ea08a\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") "
Oct 30 21:39:44.025513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.025332    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-plugins-config-volume\") pod \"7b142a0f-0389-433b-8260-304cf65ea08a\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") "
Oct 30 21:39:44.025513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.025449    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-kube-api-access-hbwsm\" (UniqueName: \"kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-kube-api-access-hbwsm\") pod \"7b142a0f-0389-433b-8260-304cf65ea08a\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") "
Oct 30 21:39:44.025652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.025541    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-provider-credentials\") pod \"7b142a0f-0389-433b-8260-304cf65ea08a\" (UID: \"7b142a0f-0389-433b-8260-304cf65ea08a\") "
Oct 30 21:39:44.026819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.026788    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-plugins-config-volume" pod "7b142a0f-0389-433b-8260-304cf65ea08a" (UID: "7b142a0f-0389-433b-8260-304cf65ea08a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:44.027207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.027180    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-routing-overlay-volume" pod "7b142a0f-0389-433b-8260-304cf65ea08a" (UID: "7b142a0f-0389-433b-8260-304cf65ea08a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:44.028910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.028872    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-provider-credentials" pod "7b142a0f-0389-433b-8260-304cf65ea08a" (UID: "7b142a0f-0389-433b-8260-304cf65ea08a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:44.029018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.028948    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-kube-api-access-hbwsm" pod "7b142a0f-0389-433b-8260-304cf65ea08a" (UID: "7b142a0f-0389-433b-8260-304cf65ea08a"). InnerVolumeSpecName "kube-api-access-hbwsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:44.071505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.071469    4918 generic.go:356] "Generic (PLEG): container finished" podID="7b142a0f-0389-433b-8260-304cf65ea08a" containerID="e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5" exitCode=0
Oct 30 21:39:44.071682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.071513    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"
Oct 30 21:39:44.071768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.071718    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw" event={"ID":"7b142a0f-0389-433b-8260-304cf65ea08a","Type":"ContainerDied","Data":"e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5"}
Oct 30 21:39:44.071833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.071765    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw" event={"ID":"7b142a0f-0389-433b-8260-304cf65ea08a","Type":"ContainerDied","Data":"682a0f0c09ebdf88b6b71d1f8d4d8672c1a34dbe6f09dd71a5fed9bc26658916"}
Oct 30 21:39:44.071833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.071795    4918 scope.go:122] "RemoveContainer" containerID="e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5"
Oct 30 21:39:44.094839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.094716    4918 scope.go:122] "RemoveContainer" containerID="e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5"
Oct 30 21:39:44.095619 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:44.095587    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5\": container with ID starting with e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5 not found: ID does not exist" containerID="e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5"
Oct 30 21:39:44.095812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.095782    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5"} err="failed to get container status \"e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5\": rpc error: code = NotFound desc = could not find container \"e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5\": container with ID starting with e4b1237b59df4d0468ad052ac89b159f9d067d55bd3983dee88091e178815ee5 not found: ID does not exist"
Oct 30 21:39:44.115957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.115920    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"]
Oct 30 21:39:44.121226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.121203    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-68c55b6465-6g2zw"]
Oct 30 21:39:44.128095 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.128066    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:44.128216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.128099    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b142a0f-0389-433b-8260-304cf65ea08a-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:44.128216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.128115    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbwsm\" (UniqueName: \"kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-kube-api-access-hbwsm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:44.128216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:44.128130    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b142a0f-0389-433b-8260-304cf65ea08a-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:45.080587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.080550    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b142a0f-0389-433b-8260-304cf65ea08a" path="/var/lib/kubelet/pods/7b142a0f-0389-433b-8260-304cf65ea08a/volumes"
Oct 30 21:39:45.805008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.804960    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"]
Oct 30 21:39:45.805924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.805907    4918 cpu_manager.go:414] "containerMap: removing container" podUID="51190d58-33e9-49de-b6ce-4086f1e48254" containerName="revoke-keys"
Oct 30 21:39:45.805983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.805926    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51190d58-33e9-49de-b6ce-4086f1e48254" containerName="revoke-keys"
Oct 30 21:39:45.805983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.805946    4918 cpu_manager.go:414] "containerMap: removing container" podUID="7b142a0f-0389-433b-8260-304cf65ea08a" containerName="payload-processing"
Oct 30 21:39:45.805983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.805951    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b142a0f-0389-433b-8260-304cf65ea08a" containerName="payload-processing"
Oct 30 21:39:45.805983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.805978    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6def2c6d-d2fa-4f90-8dcc-4e2a2507e956" containerName="revoke-keys"
Oct 30 21:39:45.805983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.805983    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6def2c6d-d2fa-4f90-8dcc-4e2a2507e956" containerName="revoke-keys"
Oct 30 21:39:45.806144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.806110    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="51190d58-33e9-49de-b6ce-4086f1e48254" containerName="revoke-keys"
Oct 30 21:39:45.806144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.806123    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b142a0f-0389-433b-8260-304cf65ea08a" containerName="payload-processing"
Oct 30 21:39:45.806144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.806133    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6def2c6d-d2fa-4f90-8dcc-4e2a2507e956" containerName="revoke-keys"
Oct 30 21:39:45.811663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.811638    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:45.817533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.817318    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:39:45.817533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.817459    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:39:45.817736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.817581    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:45.817736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.817613    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:45.819598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.819576    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"]
Oct 30 21:39:45.863571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.863535    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20740d0-j2pff\" (UID: \"51cb2578-9925-448f-b941-0fc2c8991373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:45.863937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.863911    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20740d0-j2pff\" (UID: \"51cb2578-9925-448f-b941-0fc2c8991373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:45.864117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.864092    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7kc5\" (UniqueName: \"kubernetes.io/projected/51cb2578-9925-448f-b941-0fc2c8991373-kube-api-access-m7kc5\") pod \"maas-api-revoke-sub-f20740d0-j2pff\" (UID: \"51cb2578-9925-448f-b941-0fc2c8991373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:45.965769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.965729    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20740d0-j2pff\" (UID: \"51cb2578-9925-448f-b941-0fc2c8991373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:45.965961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.965900    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20740d0-j2pff\" (UID: \"51cb2578-9925-448f-b941-0fc2c8991373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:45.966100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.966083    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7kc5\" (UniqueName: \"kubernetes.io/projected/51cb2578-9925-448f-b941-0fc2c8991373-kube-api-access-m7kc5\") pod \"maas-api-revoke-sub-f20740d0-j2pff\" (UID: \"51cb2578-9925-448f-b941-0fc2c8991373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:45.966651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.966628    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f20740d0-j2pff\" (UID: \"51cb2578-9925-448f-b941-0fc2c8991373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:45.968317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.968299    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f20740d0-j2pff\" (UID: \"51cb2578-9925-448f-b941-0fc2c8991373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:45.974863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:45.974831    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7kc5\" (UniqueName: \"kubernetes.io/projected/51cb2578-9925-448f-b941-0fc2c8991373-kube-api-access-m7kc5\") pod \"maas-api-revoke-sub-f20740d0-j2pff\" (UID: \"51cb2578-9925-448f-b941-0fc2c8991373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:46.128824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:46.128743    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:46.448306 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:39:46.448273    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cb2578_9925_448f_b941_0fc2c8991373.slice/crio-4901aaf488a5efdbe8e28cae07e147eba785acf0ae9765076b6737f3cd5be9dd WatchSource:0}: Error finding container 4901aaf488a5efdbe8e28cae07e147eba785acf0ae9765076b6737f3cd5be9dd: Status 404 returned error can't find the container with id 4901aaf488a5efdbe8e28cae07e147eba785acf0ae9765076b6737f3cd5be9dd
Oct 30 21:39:46.449675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:46.449650    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"]
Oct 30 21:39:47.167097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.167006    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff" event={"ID":"51cb2578-9925-448f-b941-0fc2c8991373","Type":"ContainerStarted","Data":"2af7c8a6bad9f1a87862179fc773e3de8787aec7ad319d5e6617349c0e23afe7"}
Oct 30 21:39:47.167097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.167065    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff" event={"ID":"51cb2578-9925-448f-b941-0fc2c8991373","Type":"ContainerStarted","Data":"4901aaf488a5efdbe8e28cae07e147eba785acf0ae9765076b6737f3cd5be9dd"}
Oct 30 21:39:47.195661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.195603    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff" podStartSLOduration=2.195546935 podStartE2EDuration="2.195546935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:47.190571228 +0000 UTC m=+2300.237130394" watchObservedRunningTime="2026-10-30 21:39:47.195546935 +0000 UTC m=+2300.242106080"
Oct 30 21:39:47.276718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.276682    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"]
Oct 30 21:39:47.282544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.282519    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.295919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.295889    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"]
Oct 30 21:39:47.305018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.304982    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.305195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.305119    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fed745f-0165-449e-97fd-5549798ae0fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.305284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.305264    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.305382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.305360    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.305514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.305495    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.305579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.305548    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d7hh\" (UniqueName: \"kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-kube-api-access-8d7hh\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.305690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.305670    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fed745f-0165-449e-97fd-5549798ae0fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.305901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.305875    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.305971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.305921    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.329024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.328745    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"]
Oct 30 21:39:47.340035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.339989    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"]
Oct 30 21:39:47.340234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.340096    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.407603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.407563    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.407793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.407659    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.407793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.407706    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fed745f-0165-449e-97fd-5549798ae0fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.407793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.407746    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.408010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.407829    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.408010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.407858    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.408010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.407900    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.408010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.407942    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.408010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.407983    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d7hh\" (UniqueName: \"kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-kube-api-access-8d7hh\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.408288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.408030    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.408288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.408078    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fed745f-0165-449e-97fd-5549798ae0fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.408288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.408131    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.408288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.408153    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82bc3cc6-e178-40d5-a7f0-240760e37408-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.408288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.408209    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpj4z\" (UniqueName: \"kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-kube-api-access-cpj4z\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.408288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.408281    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.408655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.408311    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.408655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.408338    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.408655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.408409    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.409764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.409734    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fed745f-0165-449e-97fd-5549798ae0fc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.410083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.410047    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.410301 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.410278    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.410744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.410720    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.410978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.410957    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.414648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.414629    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fed745f-0165-449e-97fd-5549798ae0fc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.416761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.416286    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"]
Oct 30 21:39:47.424679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.424622    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:47.428797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.428654    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.431756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.431733    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d7hh\" (UniqueName: \"kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-kube-api-access-8d7hh\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.432588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.432562    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-zf2bd\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.436400 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.436357    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"]
Oct 30 21:39:47.509622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.509589    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpj4z\" (UniqueName: \"kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-kube-api-access-cpj4z\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.509740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.509665    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.509740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.509698    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4htj\" (UniqueName: \"kubernetes.io/projected/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-kube-api-access-w4htj\") pod \"maas-api-revoke-sub-1cce0ace-p54ws\" (UID: \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:47.509740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.509722    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.509940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.509915    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.510018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.510001    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.510112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.510092    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.510112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.510101    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.510316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.510229    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.510316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.510282    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cce0ace-p54ws\" (UID: \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:47.510316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.510296    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.510520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.510335    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.510520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.510354    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.510520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.510448    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82bc3cc6-e178-40d5-a7f0-240760e37408-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.510701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.510623    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cce0ace-p54ws\" (UID: \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:47.510701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.510630    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.511060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.511042    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82bc3cc6-e178-40d5-a7f0-240760e37408-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.513149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.513018    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.513621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.513605    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.517747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.517727    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpj4z\" (UniqueName: \"kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-kube-api-access-cpj4z\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.518157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.518135    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-h7txm\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.604758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.604731    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:47.611753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.611733    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cce0ace-p54ws\" (UID: \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:47.611873 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.611777    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cce0ace-p54ws\" (UID: \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:47.611873 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.611843    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4htj\" (UniqueName: \"kubernetes.io/projected/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-kube-api-access-w4htj\") pod \"maas-api-revoke-sub-1cce0ace-p54ws\" (UID: \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:47.612363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.612341    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cce0ace-p54ws\" (UID: \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:47.614341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.614317    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cce0ace-p54ws\" (UID: \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:47.620553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.620535    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4htj\" (UniqueName: \"kubernetes.io/projected/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-kube-api-access-w4htj\") pod \"maas-api-revoke-sub-1cce0ace-p54ws\" (UID: \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:47.657822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.657765    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:47.769742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.769713    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:47.898496 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:39:47.896639    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fed745f_0165_449e_97fd_5549798ae0fc.slice/crio-2e5a93ad355b304bcf4bdb70ee9a6d0f43a48666e6779f8ad930dd87144e840b WatchSource:0}: Error finding container 2e5a93ad355b304bcf4bdb70ee9a6d0f43a48666e6779f8ad930dd87144e840b: Status 404 returned error can't find the container with id 2e5a93ad355b304bcf4bdb70ee9a6d0f43a48666e6779f8ad930dd87144e840b
Oct 30 21:39:47.911547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.910567    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:39:47.911547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.910726    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:39:47.911547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.910813    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:39:47.919663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:47.918099    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"]
Oct 30 21:39:48.125995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:48.115852    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:39:48.125995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:48.115934    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:39:48.125995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:48.115974    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:39:48.136841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:48.136815    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"]
Oct 30 21:39:48.145839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:48.145795    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"]
Oct 30 21:39:48.184155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:48.184124    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" event={"ID":"82bc3cc6-e178-40d5-a7f0-240760e37408","Type":"ContainerStarted","Data":"8137bc26ddbbddf5dfc21c2a727ef55cc483af25aa8b3f6e8dd91b8528e3870c"}
Oct 30 21:39:48.185730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:48.185697    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws" event={"ID":"aabba317-ead4-4fac-ab08-a6f16eb9bd5b","Type":"ContainerStarted","Data":"e00a41d54257508b01bf08d6c5e2d9820005db5ff2086dc65e4df21bd57b7aa5"}
Oct 30 21:39:48.187235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:48.187208    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" event={"ID":"5fed745f-0165-449e-97fd-5549798ae0fc","Type":"ContainerStarted","Data":"2e5a93ad355b304bcf4bdb70ee9a6d0f43a48666e6779f8ad930dd87144e840b"}
Oct 30 21:39:48.189104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:48.189082    4918 generic.go:356] "Generic (PLEG): container finished" podID="51cb2578-9925-448f-b941-0fc2c8991373" containerID="2af7c8a6bad9f1a87862179fc773e3de8787aec7ad319d5e6617349c0e23afe7" exitCode=0
Oct 30 21:39:48.189213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:48.189192    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff" event={"ID":"51cb2578-9925-448f-b941-0fc2c8991373","Type":"ContainerDied","Data":"2af7c8a6bad9f1a87862179fc773e3de8787aec7ad319d5e6617349c0e23afe7"}
Oct 30 21:39:49.256009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.255972    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws" event={"ID":"aabba317-ead4-4fac-ab08-a6f16eb9bd5b","Type":"ContainerStarted","Data":"4cd16d7a100c885a20154070e25002b6abe464ed6ee9c5553f1c2b52cb65b0d4"}
Oct 30 21:39:49.258007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.257978    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" event={"ID":"5fed745f-0165-449e-97fd-5549798ae0fc","Type":"ContainerStarted","Data":"615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793"}
Oct 30 21:39:49.260242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.260219    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" event={"ID":"82bc3cc6-e178-40d5-a7f0-240760e37408","Type":"ContainerStarted","Data":"9b8af7dfca8f02bf604c4d6af306d2fc260e4f7982bc7831f924e26af18030ea"}
Oct 30 21:39:49.284457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.284364    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws" podStartSLOduration=2.284351403 podStartE2EDuration="2.284351403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:49.278666884 +0000 UTC m=+2302.325226047" watchObservedRunningTime="2026-10-30 21:39:49.284351403 +0000 UTC m=+2302.330910548"
Oct 30 21:39:49.346773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.346719    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" podStartSLOduration=2.346704076 podStartE2EDuration="2.346704076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:49.312421145 +0000 UTC m=+2302.358980289" watchObservedRunningTime="2026-10-30 21:39:49.346704076 +0000 UTC m=+2302.393263221"
Oct 30 21:39:49.367557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.367500    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" podStartSLOduration=2.367484956 podStartE2EDuration="2.367484956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:49.361886193 +0000 UTC m=+2302.408445357" watchObservedRunningTime="2026-10-30 21:39:49.367484956 +0000 UTC m=+2302.414044101"
Oct 30 21:39:49.650851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.620072    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:49.674329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.652854    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zf2bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body=
Oct 30 21:39:49.674329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.652914    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused"
Oct 30 21:39:49.720139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.698945    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:49.727812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.726532    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h7txm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body=
Oct 30 21:39:49.727812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.726637    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused"
Oct 30 21:39:49.751291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.751159    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:49.882157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.881895    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-xks-ca\") pod \"51cb2578-9925-448f-b941-0fc2c8991373\" (UID: \"51cb2578-9925-448f-b941-0fc2c8991373\") "
Oct 30 21:39:49.882157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.881977    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-openshift-ca\") pod \"51cb2578-9925-448f-b941-0fc2c8991373\" (UID: \"51cb2578-9925-448f-b941-0fc2c8991373\") "
Oct 30 21:39:49.882481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.882217    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cb2578-9925-448f-b941-0fc2c8991373-kube-api-access-m7kc5\" (UniqueName: \"kubernetes.io/projected/51cb2578-9925-448f-b941-0fc2c8991373-kube-api-access-m7kc5\") pod \"51cb2578-9925-448f-b941-0fc2c8991373\" (UID: \"51cb2578-9925-448f-b941-0fc2c8991373\") "
Oct 30 21:39:49.883052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.883007    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-openshift-ca" pod "51cb2578-9925-448f-b941-0fc2c8991373" (UID: "51cb2578-9925-448f-b941-0fc2c8991373"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:49.884742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.884714    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-xks-ca" pod "51cb2578-9925-448f-b941-0fc2c8991373" (UID: "51cb2578-9925-448f-b941-0fc2c8991373"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:49.944888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.944596    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cb2578-9925-448f-b941-0fc2c8991373-kube-api-access-m7kc5" pod "51cb2578-9925-448f-b941-0fc2c8991373" (UID: "51cb2578-9925-448f-b941-0fc2c8991373"). InnerVolumeSpecName "kube-api-access-m7kc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:49.985160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.985136    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:49.985250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.985164    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51cb2578-9925-448f-b941-0fc2c8991373-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:49.985250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:49.985174    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7kc5\" (UniqueName: \"kubernetes.io/projected/51cb2578-9925-448f-b941-0fc2c8991373-kube-api-access-m7kc5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:50.271447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:50.271411    4918 generic.go:356] "Generic (PLEG): container finished" podID="aabba317-ead4-4fac-ab08-a6f16eb9bd5b" containerID="4cd16d7a100c885a20154070e25002b6abe464ed6ee9c5553f1c2b52cb65b0d4" exitCode=0
Oct 30 21:39:50.271928 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:50.271450    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws" event={"ID":"aabba317-ead4-4fac-ab08-a6f16eb9bd5b","Type":"ContainerDied","Data":"4cd16d7a100c885a20154070e25002b6abe464ed6ee9c5553f1c2b52cb65b0d4"}
Oct 30 21:39:50.272947 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:50.272923    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff" event={"ID":"51cb2578-9925-448f-b941-0fc2c8991373","Type":"ContainerDied","Data":"4901aaf488a5efdbe8e28cae07e147eba785acf0ae9765076b6737f3cd5be9dd"}
Oct 30 21:39:50.273069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:50.272952    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"
Oct 30 21:39:50.273069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:50.272969    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4901aaf488a5efdbe8e28cae07e147eba785acf0ae9765076b6737f3cd5be9dd"
Oct 30 21:39:50.608058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:50.607974    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zf2bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body=
Oct 30 21:39:50.608328 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:50.608298    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused"
Oct 30 21:39:50.666052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:50.666016    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h7txm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body=
Oct 30 21:39:50.666350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:50.666322    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused"
Oct 30 21:39:51.641474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:51.641328    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zf2bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body=
Oct 30 21:39:51.641474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:51.641404    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused"
Oct 30 21:39:51.661772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:51.661738    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h7txm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body=
Oct 30 21:39:51.661914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:51.661796    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused"
Oct 30 21:39:51.754740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:51.754713    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:51.884339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:51.884301    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-xks-ca\") pod \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\" (UID: \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\") "
Oct 30 21:39:51.884680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:51.884658    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-openshift-ca\") pod \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\" (UID: \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\") "
Oct 30 21:39:51.884852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:51.884835    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-kube-api-access-w4htj\" (UniqueName: \"kubernetes.io/projected/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-kube-api-access-w4htj\") pod \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\" (UID: \"aabba317-ead4-4fac-ab08-a6f16eb9bd5b\") "
Oct 30 21:39:51.885105 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:51.885077    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-openshift-ca" pod "aabba317-ead4-4fac-ab08-a6f16eb9bd5b" (UID: "aabba317-ead4-4fac-ab08-a6f16eb9bd5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:51.897460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:51.885774    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:51.930837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:51.930806    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-kube-api-access-w4htj" pod "aabba317-ead4-4fac-ab08-a6f16eb9bd5b" (UID: "aabba317-ead4-4fac-ab08-a6f16eb9bd5b"). InnerVolumeSpecName "kube-api-access-w4htj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:51.943059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:51.943001    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-xks-ca" pod "aabba317-ead4-4fac-ab08-a6f16eb9bd5b" (UID: "aabba317-ead4-4fac-ab08-a6f16eb9bd5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:52.041561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:52.017328    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4htj\" (UniqueName: \"kubernetes.io/projected/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-kube-api-access-w4htj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:52.045566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:52.045539    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aabba317-ead4-4fac-ab08-a6f16eb9bd5b-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:52.344705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:52.344671    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws" event={"ID":"aabba317-ead4-4fac-ab08-a6f16eb9bd5b","Type":"ContainerDied","Data":"e00a41d54257508b01bf08d6c5e2d9820005db5ff2086dc65e4df21bd57b7aa5"}
Oct 30 21:39:52.344868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:52.344722    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e00a41d54257508b01bf08d6c5e2d9820005db5ff2086dc65e4df21bd57b7aa5"
Oct 30 21:39:52.344868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:52.344682    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"
Oct 30 21:39:52.611333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:52.610794    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zf2bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body=
Oct 30 21:39:52.611333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:52.610859    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused"
Oct 30 21:39:52.658609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:52.658574    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h7txm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body=
Oct 30 21:39:52.659082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:52.658630    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused"
Oct 30 21:39:53.611417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:53.609848    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zf2bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body=
Oct 30 21:39:53.611417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:53.609913    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused"
Oct 30 21:39:53.658549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:53.658519    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h7txm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body=
Oct 30 21:39:53.658737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:53.658570    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused"
Oct 30 21:39:54.618416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:54.607838    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zf2bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body=
Oct 30 21:39:54.618416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:54.607915    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused"
Oct 30 21:39:54.658894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:54.658852    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h7txm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body=
Oct 30 21:39:54.659415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:54.658915    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused"
Oct 30 21:39:55.605729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:55.605691    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zf2bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused" start-of-body=
Oct 30 21:39:55.605929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:55.605759    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": dial tcp 10.132.1.85:15021: connect: connection refused"
Oct 30 21:39:55.658721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:55.658683    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h7txm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body=
Oct 30 21:39:55.658876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:55.658749    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused"
Oct 30 21:39:57.026526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.015458    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"]
Oct 30 21:39:57.026526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.016828    4918 cpu_manager.go:414] "containerMap: removing container" podUID="aabba317-ead4-4fac-ab08-a6f16eb9bd5b" containerName="revoke-keys"
Oct 30 21:39:57.026526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.016851    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aabba317-ead4-4fac-ab08-a6f16eb9bd5b" containerName="revoke-keys"
Oct 30 21:39:57.026526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.016944    4918 cpu_manager.go:414] "containerMap: removing container" podUID="51cb2578-9925-448f-b941-0fc2c8991373" containerName="revoke-keys"
Oct 30 21:39:57.026526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.016953    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cb2578-9925-448f-b941-0fc2c8991373" containerName="revoke-keys"
Oct 30 21:39:57.026526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.017117    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cb2578-9925-448f-b941-0fc2c8991373" containerName="revoke-keys"
Oct 30 21:39:57.026526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.017137    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="aabba317-ead4-4fac-ab08-a6f16eb9bd5b" containerName="revoke-keys"
Oct 30 21:39:57.035506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.035476    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.057415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.053794    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"payload-processing-external-model-plugins-e2e-extmod-a-729802\""
Oct 30 21:39:57.058031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.057917    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"routing-overlay\""
Oct 30 21:39:57.061951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.061930    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"kube-root-ca.crt\""
Oct 30 21:39:57.062786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.062762    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"payload-processing-external-model-e2e--42f38b37-dockercfg-6wsvt\""
Oct 30 21:39:57.062989 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.062973    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"agc-cred-b22596d4938d\""
Oct 30 21:39:57.063184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.063147    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-729802\"/\"openshift-service-ca.crt\""
Oct 30 21:39:57.165414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.109974    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.165414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.144632    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdvtm\" (UniqueName: \"kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-kube-api-access-vdvtm\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.165414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.144782    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.165414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.144827    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.167369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.167323    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"]
Oct 30 21:39:57.312469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.247155    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.315841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.315809    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdvtm\" (UniqueName: \"kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-kube-api-access-vdvtm\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.316235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.316216    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.317248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.317214    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.318922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.317129    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.319465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.318207    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.321120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.321089    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.327907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.325319    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdvtm\" (UniqueName: \"kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-kube-api-access-vdvtm\") pod \"payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") " pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.409534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.409504    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:57.651676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.651560    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-zf2bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:39:57.651676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.651642    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:57.705026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.704419    4918 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-h7txm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:39:57.705026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.704479    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:39:57.818822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.818785    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"]
Oct 30 21:39:57.825123 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:39:57.825096    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60e9a02_d0a5_47eb_965f_7df22fa20f76.slice/crio-980e5b9a8ab6a7c800b3e629764da78f88ee1d5f890932d7256ef7d0a622fb87 WatchSource:0}: Error finding container 980e5b9a8ab6a7c800b3e629764da78f88ee1d5f890932d7256ef7d0a622fb87: Status 404 returned error can't find the container with id 980e5b9a8ab6a7c800b3e629764da78f88ee1d5f890932d7256ef7d0a622fb87
Oct 30 21:39:57.902025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.901995    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"]
Oct 30 21:39:57.968502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.968471    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:57.971988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.971964    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:57.972265 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.972222    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:57.973497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.973473    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:57.975492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.975465    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:39:57.979712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:57.979458    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:39:58.489504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:58.485572    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd" event={"ID":"d60e9a02-d0a5-47eb-965f-7df22fa20f76","Type":"ContainerStarted","Data":"6b87aad91f64c4c6ee8add8ab5e0c3ee79d83e57588d8998f2f2df304b2a538b"}
Oct 30 21:39:58.489504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:58.485631    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd" event={"ID":"d60e9a02-d0a5-47eb-965f-7df22fa20f76","Type":"ContainerStarted","Data":"980e5b9a8ab6a7c800b3e629764da78f88ee1d5f890932d7256ef7d0a622fb87"}
Oct 30 21:39:58.489504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:58.486678    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:39:58.592614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:39:58.592542    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd" podStartSLOduration=2.592518115 podStartE2EDuration="2.592518115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:58.538402917 +0000 UTC m=+2311.584962101" watchObservedRunningTime="2026-10-30 21:39:58.592518115 +0000 UTC m=+2311.639077260"
Oct 30 21:39:59.168896 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:39:59.168864    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:40:03.486514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:03.486465    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd" podUID="d60e9a02-d0a5-47eb-965f-7df22fa20f76" containerName="payload-processing" containerID="cri-o://6b87aad91f64c4c6ee8add8ab5e0c3ee79d83e57588d8998f2f2df304b2a538b" gracePeriod=25
Oct 30 21:40:03.607023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:03.606984    4918 generic.go:356] "Generic (PLEG): container finished" podID="d60e9a02-d0a5-47eb-965f-7df22fa20f76" containerID="6b87aad91f64c4c6ee8add8ab5e0c3ee79d83e57588d8998f2f2df304b2a538b" exitCode=0
Oct 30 21:40:03.607177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:03.607051    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd" event={"ID":"d60e9a02-d0a5-47eb-965f-7df22fa20f76","Type":"ContainerDied","Data":"6b87aad91f64c4c6ee8add8ab5e0c3ee79d83e57588d8998f2f2df304b2a538b"}
Oct 30 21:40:03.803037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:03.803012    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:40:03.875425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:03.872443    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-plugins-config-volume\") pod \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") "
Oct 30 21:40:03.875425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:03.872707    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-kube-api-access-vdvtm\" (UniqueName: \"kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-kube-api-access-vdvtm\") pod \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") "
Oct 30 21:40:03.875425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:03.872860    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-routing-overlay-volume\") pod \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") "
Oct 30 21:40:03.875425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:03.873058    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-provider-credentials\") pod \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\" (UID: \"d60e9a02-d0a5-47eb-965f-7df22fa20f76\") "
Oct 30 21:40:03.956423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:03.896559    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-routing-overlay-volume" pod "d60e9a02-d0a5-47eb-965f-7df22fa20f76" (UID: "d60e9a02-d0a5-47eb-965f-7df22fa20f76"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:03.956423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:03.896978    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-plugins-config-volume" pod "d60e9a02-d0a5-47eb-965f-7df22fa20f76" (UID: "d60e9a02-d0a5-47eb-965f-7df22fa20f76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:04.034465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:04.021450    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:04.034465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:04.021489    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d60e9a02-d0a5-47eb-965f-7df22fa20f76-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:04.055690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:04.055651    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-kube-api-access-vdvtm" pod "d60e9a02-d0a5-47eb-965f-7df22fa20f76" (UID: "d60e9a02-d0a5-47eb-965f-7df22fa20f76"). InnerVolumeSpecName "kube-api-access-vdvtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:04.131830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:04.102787    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-provider-credentials" pod "d60e9a02-d0a5-47eb-965f-7df22fa20f76" (UID: "d60e9a02-d0a5-47eb-965f-7df22fa20f76"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:04.276117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:04.276074    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:04.278512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:04.278485    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdvtm\" (UniqueName: \"kubernetes.io/projected/d60e9a02-d0a5-47eb-965f-7df22fa20f76-kube-api-access-vdvtm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:04.617921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:04.617897    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"
Oct 30 21:40:04.618376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:04.617933    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd" event={"ID":"d60e9a02-d0a5-47eb-965f-7df22fa20f76","Type":"ContainerDied","Data":"980e5b9a8ab6a7c800b3e629764da78f88ee1d5f890932d7256ef7d0a622fb87"}
Oct 30 21:40:04.618376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:04.618000    4918 scope.go:122] "RemoveContainer" containerID="6b87aad91f64c4c6ee8add8ab5e0c3ee79d83e57588d8998f2f2df304b2a538b"
Oct 30 21:40:04.650490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:04.650459    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"]
Oct 30 21:40:04.655826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:04.655801    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-729802/payload-processing-external-model-e2e-extmod-a-729802-69cc7njdd"]
Oct 30 21:40:05.080320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:05.080289    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60e9a02-d0a5-47eb-965f-7df22fa20f76" path="/var/lib/kubelet/pods/d60e9a02-d0a5-47eb-965f-7df22fa20f76/volumes"
Oct 30 21:40:09.946110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:09.945515    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:40:09.957413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:09.956480    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:09.957662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:09.957634    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:40:09.958210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:09.958180    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:09.958617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:09.958593    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:40:09.988877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:09.988838    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:40:10.051513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:10.049900    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:40:10.055979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:10.055508    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:10.081981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:10.073132    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:10.226622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:10.226439    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 30 21:40:13.148753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.148722    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"]
Oct 30 21:40:13.149630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.149611    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d60e9a02-d0a5-47eb-965f-7df22fa20f76" containerName="payload-processing"
Oct 30 21:40:13.149630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.149627    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60e9a02-d0a5-47eb-965f-7df22fa20f76" containerName="payload-processing"
Oct 30 21:40:13.149842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.149741    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60e9a02-d0a5-47eb-965f-7df22fa20f76" containerName="payload-processing"
Oct 30 21:40:13.155497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.155467    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.162369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.162340    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fa4045c246e1\""
Oct 30 21:40:13.162547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.162437    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:40:13.162547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.162460    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:40:13.162547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.162533    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-76921397fa76\""
Oct 30 21:40:13.162547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.162544    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:40:13.162710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.162575    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:40:13.162710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.162586    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-jgk7c\""
Oct 30 21:40:13.162710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.162587    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5ac5f2767576\""
Oct 30 21:40:13.164220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.164200    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"]
Oct 30 21:40:13.277921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.277879    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28lk8\" (UniqueName: \"kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-kube-api-access-28lk8\") pod \"payload-processing-external-model-86f5c55f49-nxblv\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") " pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.278092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.277997    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-routing-overlay-volume\") pod \"payload-processing-external-model-86f5c55f49-nxblv\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") " pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.278092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.278018    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-provider-credentials\") pod \"payload-processing-external-model-86f5c55f49-nxblv\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") " pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.278176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.278147    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-plugins-config-volume\") pod \"payload-processing-external-model-86f5c55f49-nxblv\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") " pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.380426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.380356    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28lk8\" (UniqueName: \"kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-kube-api-access-28lk8\") pod \"payload-processing-external-model-86f5c55f49-nxblv\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") " pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.380644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.380450    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-routing-overlay-volume\") pod \"payload-processing-external-model-86f5c55f49-nxblv\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") " pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.380644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.380476    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-provider-credentials\") pod \"payload-processing-external-model-86f5c55f49-nxblv\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") " pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.380644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.380549    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-plugins-config-volume\") pod \"payload-processing-external-model-86f5c55f49-nxblv\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") " pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.381204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.381176    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-routing-overlay-volume\") pod \"payload-processing-external-model-86f5c55f49-nxblv\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") " pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.381304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.381255    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-plugins-config-volume\") pod \"payload-processing-external-model-86f5c55f49-nxblv\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") " pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.383240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.383221    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-provider-credentials\") pod \"payload-processing-external-model-86f5c55f49-nxblv\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") " pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.390069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.390041    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28lk8\" (UniqueName: \"kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-kube-api-access-28lk8\") pod \"payload-processing-external-model-86f5c55f49-nxblv\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") " pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:13.473008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:13.472976    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:14.138068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:14.131941    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"]
Oct 30 21:40:15.282413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:15.273668    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv" event={"ID":"f29441da-316c-4a9c-9fda-db62b0ff8cc9","Type":"ContainerStarted","Data":"4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b"}
Oct 30 21:40:15.282413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:15.273760    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv" event={"ID":"f29441da-316c-4a9c-9fda-db62b0ff8cc9","Type":"ContainerStarted","Data":"9cac2ac9a2201a921b18295e7126d4890c3a3ca702091fc5b95ecf6f1217f1c9"}
Oct 30 21:40:15.299416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:15.291020    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:16.146795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.146742    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv" podStartSLOduration=3.1467281480000002 podStartE2EDuration="3.146728148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:15.332677685 +0000 UTC m=+2328.379236833" watchObservedRunningTime="2026-10-30 21:40:16.146728148 +0000 UTC m=+2329.193287308"
Oct 30 21:40:16.151458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.151433    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"]
Oct 30 21:40:16.156660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.156641    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.162899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.162876    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"kube-root-ca.crt\""
Oct 30 21:40:16.162996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.162934    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"routing-overlay\""
Oct 30 21:40:16.162996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.162963    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8072a0\""
Oct 30 21:40:16.163081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.163008    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"openshift-service-ca.crt\""
Oct 30 21:40:16.165992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.165967    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"]
Oct 30 21:40:16.178009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.177968    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"agc-cred-4099299197dc\""
Oct 30 21:40:16.181667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.181638    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"payload-processing-external-model-e2e--74dc6735-dockercfg-vqqdw\""
Oct 30 21:40:16.252158 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.252128    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk4p5\" (UniqueName: \"kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-kube-api-access-kk4p5\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.252331 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.252309    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.252507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.252490    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.252681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.252664    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.355437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.354638    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk4p5\" (UniqueName: \"kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-kube-api-access-kk4p5\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.355437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.354825    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.355437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.354967    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.355437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.355180    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.360412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.356859    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.360412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.357176    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.387534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.380959    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.414460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.411329    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk4p5\" (UniqueName: \"kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-kube-api-access-kk4p5\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:16.476421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:16.475719    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:17.035382 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:40:17.035353    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f91907_7b5d_4a7a_9912_5d9a121c7be1.slice/crio-e4cefbbfaad6aff3020848ddd7b5cf5abbd578c1a0de2532029804c9c9002b4a WatchSource:0}: Error finding container e4cefbbfaad6aff3020848ddd7b5cf5abbd578c1a0de2532029804c9c9002b4a: Status 404 returned error can't find the container with id e4cefbbfaad6aff3020848ddd7b5cf5abbd578c1a0de2532029804c9c9002b4a
Oct 30 21:40:17.037439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:17.037412    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"]
Oct 30 21:40:17.294761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:17.294725    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" event={"ID":"37f91907-7b5d-4a7a-9912-5d9a121c7be1","Type":"ContainerStarted","Data":"1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b"}
Oct 30 21:40:17.294941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:17.294785    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" event={"ID":"37f91907-7b5d-4a7a-9912-5d9a121c7be1","Type":"ContainerStarted","Data":"e4cefbbfaad6aff3020848ddd7b5cf5abbd578c1a0de2532029804c9c9002b4a"}
Oct 30 21:40:17.294941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:17.294811    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:17.314720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:17.314649    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" podStartSLOduration=1.314635237 podStartE2EDuration="1.314635237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:17.312573538 +0000 UTC m=+2330.359132685" watchObservedRunningTime="2026-10-30 21:40:17.314635237 +0000 UTC m=+2330.361194382"
Oct 30 21:40:20.028975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:20.028929    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"]
Oct 30 21:40:20.034793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:20.034759    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f088835f-j599x"]
Oct 30 21:40:21.081836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:21.081795    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="410f3d85-4a38-40fc-b478-85e06686f9b6" path="/var/lib/kubelet/pods/410f3d85-4a38-40fc-b478-85e06686f9b6/volumes"
Oct 30 21:40:27.303338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:27.303307    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:40:28.306884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:28.306858    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:40:29.162975 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:40:29.162939    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:40:32.899422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:32.898703    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:40:32.924417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:32.917196    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:32.954413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:32.953598    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:40:32.977415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:32.968354    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:32.977415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:32.972525    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:40:32.977415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:32.972807    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:40:32.995423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:32.991465    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:33.043832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:33.043765    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:40:33.091423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:40:33.089425    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:40:59.162800 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:40:59.162750    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:41:03.679592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.679551    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"]
Oct 30 21:41:03.686636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.686616    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:03.686887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.686697    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"]
Oct 30 21:41:03.690841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.690821    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:41:03.690969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.690940    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:41:03.692037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.692018    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:41:03.692136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.692096    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:41:03.841603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.841543    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c4f40c-mbz88\" (UID: \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:03.841887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.841862    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89hg9\" (UniqueName: \"kubernetes.io/projected/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-kube-api-access-89hg9\") pod \"maas-api-revoke-sub-96c4f40c-mbz88\" (UID: \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:03.842009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.841993    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c4f40c-mbz88\" (UID: \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:03.943854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.943760    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89hg9\" (UniqueName: \"kubernetes.io/projected/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-kube-api-access-89hg9\") pod \"maas-api-revoke-sub-96c4f40c-mbz88\" (UID: \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:03.944039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.943887    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c4f40c-mbz88\" (UID: \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:03.944113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.944040    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c4f40c-mbz88\" (UID: \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:03.944579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.944559    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c4f40c-mbz88\" (UID: \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:03.946867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.946836    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c4f40c-mbz88\" (UID: \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:03.952583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:03.952557    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89hg9\" (UniqueName: \"kubernetes.io/projected/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-kube-api-access-89hg9\") pod \"maas-api-revoke-sub-96c4f40c-mbz88\" (UID: \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:04.024404 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:04.024362    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:04.312026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:41:04.311991    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a7cd32_62c7_4d7e_a67f_eb8da6cc98c3.slice/crio-05cd3fbc4d51abe3637885cd729917335d5d9a992e2f19ddb5ad322608225733 WatchSource:0}: Error finding container 05cd3fbc4d51abe3637885cd729917335d5d9a992e2f19ddb5ad322608225733: Status 404 returned error can't find the container with id 05cd3fbc4d51abe3637885cd729917335d5d9a992e2f19ddb5ad322608225733
Oct 30 21:41:04.312954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:04.312931    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"]
Oct 30 21:41:04.475196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:04.475162    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88" event={"ID":"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3","Type":"ContainerStarted","Data":"05cd3fbc4d51abe3637885cd729917335d5d9a992e2f19ddb5ad322608225733"}
Oct 30 21:41:05.485645 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:05.485608    4918 generic.go:356] "Generic (PLEG): container finished" podID="49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3" containerID="421037542ee20ce90506c74e4dfaa8d47d8ff9dc18f160784e060251d6d9dc3e" exitCode=0
Oct 30 21:41:05.486045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:05.485688    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88" event={"ID":"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3","Type":"ContainerDied","Data":"421037542ee20ce90506c74e4dfaa8d47d8ff9dc18f160784e060251d6d9dc3e"}
Oct 30 21:41:06.769505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:06.769447    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:06.820006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:06.819970    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-xks-ca\") pod \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\" (UID: \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\") "
Oct 30 21:41:06.820198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:06.820184    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-openshift-ca\") pod \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\" (UID: \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\") "
Oct 30 21:41:06.820376 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:06.820358    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-kube-api-access-89hg9\" (UniqueName: \"kubernetes.io/projected/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-kube-api-access-89hg9\") pod \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\" (UID: \"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3\") "
Oct 30 21:41:06.820646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:06.820616    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-openshift-ca" pod "49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3" (UID: "49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:06.821537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:06.821514    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:06.822775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:06.822748    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-kube-api-access-89hg9" pod "49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3" (UID: "49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3"). InnerVolumeSpecName "kube-api-access-89hg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:06.822910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:06.822893    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-xks-ca" pod "49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3" (UID: "49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:41:06.923872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:06.923836    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89hg9\" (UniqueName: \"kubernetes.io/projected/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-kube-api-access-89hg9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:06.923872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:06.923865    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:07.505415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:07.505375    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"
Oct 30 21:41:07.505616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:07.505376    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88" event={"ID":"49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3","Type":"ContainerDied","Data":"05cd3fbc4d51abe3637885cd729917335d5d9a992e2f19ddb5ad322608225733"}
Oct 30 21:41:07.505616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:07.505507    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05cd3fbc4d51abe3637885cd729917335d5d9a992e2f19ddb5ad322608225733"
Oct 30 21:41:14.256678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.256648    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"]
Oct 30 21:41:14.257537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.257522    4918 cpu_manager.go:414] "containerMap: removing container" podUID="49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3" containerName="revoke-keys"
Oct 30 21:41:14.257537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.257538    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3" containerName="revoke-keys"
Oct 30 21:41:14.257749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.257738    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3" containerName="revoke-keys"
Oct 30 21:41:14.262089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.262065    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:14.267873 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.267719    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:41:14.268606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.268584    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:41:14.268728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.268630    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:41:14.268794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.268730    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:41:14.276207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.275880    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"]
Oct 30 21:41:14.386728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.386697    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdbhd\" (UniqueName: \"kubernetes.io/projected/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-kube-api-access-tdbhd\") pod \"maas-api-revoke-sub-2cfe707f-l798j\" (UID: \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:14.386903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.386743    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cfe707f-l798j\" (UID: \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:14.386952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.386918    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cfe707f-l798j\" (UID: \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:14.488870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.488766    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdbhd\" (UniqueName: \"kubernetes.io/projected/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-kube-api-access-tdbhd\") pod \"maas-api-revoke-sub-2cfe707f-l798j\" (UID: \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:14.489092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.488898    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cfe707f-l798j\" (UID: \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:14.489092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.489045    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cfe707f-l798j\" (UID: \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:14.489740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.489717    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cfe707f-l798j\" (UID: \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:14.491596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.491572    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cfe707f-l798j\" (UID: \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:14.499485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.499460    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdbhd\" (UniqueName: \"kubernetes.io/projected/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-kube-api-access-tdbhd\") pod \"maas-api-revoke-sub-2cfe707f-l798j\" (UID: \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:14.594375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.594300    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:14.884183 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:41:14.884151    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3491ef_8d83_4636_8f33_9e7cbbd00afe.slice/crio-50a7a96b8f36396fe10048a08aaa8d3a0d1a33447d0ea6e351be0c7121f7ad17 WatchSource:0}: Error finding container 50a7a96b8f36396fe10048a08aaa8d3a0d1a33447d0ea6e351be0c7121f7ad17: Status 404 returned error can't find the container with id 50a7a96b8f36396fe10048a08aaa8d3a0d1a33447d0ea6e351be0c7121f7ad17
Oct 30 21:41:14.887222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:14.887197    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"]
Oct 30 21:41:15.701146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:15.701100    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j" event={"ID":"cb3491ef-8d83-4636-8f33-9e7cbbd00afe","Type":"ContainerStarted","Data":"6fab9515308dda326ad38d9a5d6688f09b141d79807f9760530b093933f0ddfc"}
Oct 30 21:41:15.701793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:15.701767    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j" event={"ID":"cb3491ef-8d83-4636-8f33-9e7cbbd00afe","Type":"ContainerStarted","Data":"50a7a96b8f36396fe10048a08aaa8d3a0d1a33447d0ea6e351be0c7121f7ad17"}
Oct 30 21:41:15.721750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:15.721697    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j" podStartSLOduration=1.7216804300000002 podStartE2EDuration="1.72168043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:15.7182709 +0000 UTC m=+2388.764830062" watchObservedRunningTime="2026-10-30 21:41:15.72168043 +0000 UTC m=+2388.768239575"
Oct 30 21:41:16.711987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:16.711953    4918 generic.go:356] "Generic (PLEG): container finished" podID="cb3491ef-8d83-4636-8f33-9e7cbbd00afe" containerID="6fab9515308dda326ad38d9a5d6688f09b141d79807f9760530b093933f0ddfc" exitCode=0
Oct 30 21:41:16.712471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:16.712051    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j" event={"ID":"cb3491ef-8d83-4636-8f33-9e7cbbd00afe","Type":"ContainerDied","Data":"6fab9515308dda326ad38d9a5d6688f09b141d79807f9760530b093933f0ddfc"}
Oct 30 21:41:18.095498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.095410    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:18.243770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.243735    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-xks-ca\") pod \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\" (UID: \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\") "
Oct 30 21:41:18.244522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.244497    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-kube-api-access-tdbhd\" (UniqueName: \"kubernetes.io/projected/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-kube-api-access-tdbhd\") pod \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\" (UID: \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\") "
Oct 30 21:41:18.244772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.244754    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-openshift-ca\") pod \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\" (UID: \"cb3491ef-8d83-4636-8f33-9e7cbbd00afe\") "
Oct 30 21:41:18.250887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.250851    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-openshift-ca" pod "cb3491ef-8d83-4636-8f33-9e7cbbd00afe" (UID: "cb3491ef-8d83-4636-8f33-9e7cbbd00afe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:18.267687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.267628    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-xks-ca" pod "cb3491ef-8d83-4636-8f33-9e7cbbd00afe" (UID: "cb3491ef-8d83-4636-8f33-9e7cbbd00afe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:41:18.270568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.270530    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-kube-api-access-tdbhd" pod "cb3491ef-8d83-4636-8f33-9e7cbbd00afe" (UID: "cb3491ef-8d83-4636-8f33-9e7cbbd00afe"). InnerVolumeSpecName "kube-api-access-tdbhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:18.348881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.348780    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdbhd\" (UniqueName: \"kubernetes.io/projected/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-kube-api-access-tdbhd\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:18.348881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.348824    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:18.348881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.348841    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb3491ef-8d83-4636-8f33-9e7cbbd00afe-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:18.880417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.879978    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j" event={"ID":"cb3491ef-8d83-4636-8f33-9e7cbbd00afe","Type":"ContainerDied","Data":"50a7a96b8f36396fe10048a08aaa8d3a0d1a33447d0ea6e351be0c7121f7ad17"}
Oct 30 21:41:18.880417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.880044    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50a7a96b8f36396fe10048a08aaa8d3a0d1a33447d0ea6e351be0c7121f7ad17"
Oct 30 21:41:18.880417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:18.880122    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"
Oct 30 21:41:19.907967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:19.907924    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:41:19.909700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:19.909662    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:41:19.963414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:19.962963    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:41:20.062878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:20.062835    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:41:20.530415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:20.528418    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:41:21.534822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.534786    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"]
Oct 30 21:41:21.535684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.535668    4918 cpu_manager.go:414] "containerMap: removing container" podUID="cb3491ef-8d83-4636-8f33-9e7cbbd00afe" containerName="revoke-keys"
Oct 30 21:41:21.535746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.535685    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3491ef-8d83-4636-8f33-9e7cbbd00afe" containerName="revoke-keys"
Oct 30 21:41:21.535863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.535853    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3491ef-8d83-4636-8f33-9e7cbbd00afe" containerName="revoke-keys"
Oct 30 21:41:21.540084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.540064    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:21.546464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.546441    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:41:21.546464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.546441    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:41:21.546644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.546466    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:41:21.546644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.546517    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:41:21.550416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.550372    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"]
Oct 30 21:41:21.615056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.615027    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7354c004-msdhr\" (UID: \"3fd993b1-2129-44db-9269-2943acd440b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:21.615327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.615106    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fhb8\" (UniqueName: \"kubernetes.io/projected/3fd993b1-2129-44db-9269-2943acd440b7-kube-api-access-5fhb8\") pod \"maas-api-revoke-sub-7354c004-msdhr\" (UID: \"3fd993b1-2129-44db-9269-2943acd440b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:21.615471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.615343    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7354c004-msdhr\" (UID: \"3fd993b1-2129-44db-9269-2943acd440b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:21.717482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.717450    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7354c004-msdhr\" (UID: \"3fd993b1-2129-44db-9269-2943acd440b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:21.717638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.717547    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fhb8\" (UniqueName: \"kubernetes.io/projected/3fd993b1-2129-44db-9269-2943acd440b7-kube-api-access-5fhb8\") pod \"maas-api-revoke-sub-7354c004-msdhr\" (UID: \"3fd993b1-2129-44db-9269-2943acd440b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:21.717716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.717699    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7354c004-msdhr\" (UID: \"3fd993b1-2129-44db-9269-2943acd440b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:21.718379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.718356    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7354c004-msdhr\" (UID: \"3fd993b1-2129-44db-9269-2943acd440b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:21.720176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.720155    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7354c004-msdhr\" (UID: \"3fd993b1-2129-44db-9269-2943acd440b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:21.726641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.726614    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fhb8\" (UniqueName: \"kubernetes.io/projected/3fd993b1-2129-44db-9269-2943acd440b7-kube-api-access-5fhb8\") pod \"maas-api-revoke-sub-7354c004-msdhr\" (UID: \"3fd993b1-2129-44db-9269-2943acd440b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:21.860244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:21.860170    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:22.143045 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:41:22.143012    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd993b1_2129_44db_9269_2943acd440b7.slice/crio-5e7847f3f22bdeb2ee1b191428a73241a443a807b6a513b1f3ece611ec458632 WatchSource:0}: Error finding container 5e7847f3f22bdeb2ee1b191428a73241a443a807b6a513b1f3ece611ec458632: Status 404 returned error can't find the container with id 5e7847f3f22bdeb2ee1b191428a73241a443a807b6a513b1f3ece611ec458632
Oct 30 21:41:22.144652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:22.144544    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"]
Oct 30 21:41:23.038856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:23.038818    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr" event={"ID":"3fd993b1-2129-44db-9269-2943acd440b7","Type":"ContainerStarted","Data":"159853b42dd188278141b01f679abcb09186591e50a1a204ff619e3452ca6f60"}
Oct 30 21:41:23.039279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:23.038875    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr" event={"ID":"3fd993b1-2129-44db-9269-2943acd440b7","Type":"ContainerStarted","Data":"5e7847f3f22bdeb2ee1b191428a73241a443a807b6a513b1f3ece611ec458632"}
Oct 30 21:41:23.059252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:23.059202    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr" podStartSLOduration=2.05918683 podStartE2EDuration="2.05918683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:23.057000675 +0000 UTC m=+2396.103559820" watchObservedRunningTime="2026-10-30 21:41:23.05918683 +0000 UTC m=+2396.105746017"
Oct 30 21:41:24.049450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:24.049376    4918 generic.go:356] "Generic (PLEG): container finished" podID="3fd993b1-2129-44db-9269-2943acd440b7" containerID="159853b42dd188278141b01f679abcb09186591e50a1a204ff619e3452ca6f60" exitCode=0
Oct 30 21:41:24.049898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:24.049454    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr" event={"ID":"3fd993b1-2129-44db-9269-2943acd440b7","Type":"ContainerDied","Data":"159853b42dd188278141b01f679abcb09186591e50a1a204ff619e3452ca6f60"}
Oct 30 21:41:25.574480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:25.572653    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:25.651493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:25.651073    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd993b1-2129-44db-9269-2943acd440b7-kube-api-access-5fhb8\" (UniqueName: \"kubernetes.io/projected/3fd993b1-2129-44db-9269-2943acd440b7-kube-api-access-5fhb8\") pod \"3fd993b1-2129-44db-9269-2943acd440b7\" (UID: \"3fd993b1-2129-44db-9269-2943acd440b7\") "
Oct 30 21:41:25.651493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:25.651176    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-openshift-ca\") pod \"3fd993b1-2129-44db-9269-2943acd440b7\" (UID: \"3fd993b1-2129-44db-9269-2943acd440b7\") "
Oct 30 21:41:25.651493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:25.651211    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-xks-ca\") pod \"3fd993b1-2129-44db-9269-2943acd440b7\" (UID: \"3fd993b1-2129-44db-9269-2943acd440b7\") "
Oct 30 21:41:25.670010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:25.669973    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd993b1-2129-44db-9269-2943acd440b7-kube-api-access-5fhb8" pod "3fd993b1-2129-44db-9269-2943acd440b7" (UID: "3fd993b1-2129-44db-9269-2943acd440b7"). InnerVolumeSpecName "kube-api-access-5fhb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:25.670544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:25.670478    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-openshift-ca" pod "3fd993b1-2129-44db-9269-2943acd440b7" (UID: "3fd993b1-2129-44db-9269-2943acd440b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:25.674284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:25.674251    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-xks-ca" pod "3fd993b1-2129-44db-9269-2943acd440b7" (UID: "3fd993b1-2129-44db-9269-2943acd440b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:41:25.767732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:25.767691    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:25.767732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:25.767728    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd993b1-2129-44db-9269-2943acd440b7-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:25.767732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:25.767739    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fhb8\" (UniqueName: \"kubernetes.io/projected/3fd993b1-2129-44db-9269-2943acd440b7-kube-api-access-5fhb8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:26.076733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:26.076705    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"
Oct 30 21:41:26.076910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:26.076745    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr" event={"ID":"3fd993b1-2129-44db-9269-2943acd440b7","Type":"ContainerDied","Data":"5e7847f3f22bdeb2ee1b191428a73241a443a807b6a513b1f3ece611ec458632"}
Oct 30 21:41:26.076910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:26.076788    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e7847f3f22bdeb2ee1b191428a73241a443a807b6a513b1f3ece611ec458632"
Oct 30 21:41:26.417415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:26.417304    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"]
Oct 30 21:41:29.164448 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:29.164361    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:41:30.438995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:30.438963    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:41:30.441634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:30.441611    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:41:31.418469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.418428    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr" podUID="fa6e64e5-fe59-4338-bbba-7931f806abca" containerName="payload-processing" containerID="cri-o://d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a" gracePeriod=25
Oct 30 21:41:31.729729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.729697    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:41:31.847533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.847329    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-provider-credentials\") pod \"fa6e64e5-fe59-4338-bbba-7931f806abca\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") "
Oct 30 21:41:31.847972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.847946    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-routing-overlay-volume\") pod \"fa6e64e5-fe59-4338-bbba-7931f806abca\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") "
Oct 30 21:41:31.848643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.848617    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-plugins-config-volume\") pod \"fa6e64e5-fe59-4338-bbba-7931f806abca\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") "
Oct 30 21:41:31.848817 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.848802    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-kube-api-access-9kxfs\" (UniqueName: \"kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-kube-api-access-9kxfs\") pod \"fa6e64e5-fe59-4338-bbba-7931f806abca\" (UID: \"fa6e64e5-fe59-4338-bbba-7931f806abca\") "
Oct 30 21:41:31.849866 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.849831    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-routing-overlay-volume" pod "fa6e64e5-fe59-4338-bbba-7931f806abca" (UID: "fa6e64e5-fe59-4338-bbba-7931f806abca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:31.851537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.851088    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-provider-credentials" pod "fa6e64e5-fe59-4338-bbba-7931f806abca" (UID: "fa6e64e5-fe59-4338-bbba-7931f806abca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:31.860143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.860111    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-kube-api-access-9kxfs" pod "fa6e64e5-fe59-4338-bbba-7931f806abca" (UID: "fa6e64e5-fe59-4338-bbba-7931f806abca"). InnerVolumeSpecName "kube-api-access-9kxfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:31.884613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.873643    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-plugins-config-volume" pod "fa6e64e5-fe59-4338-bbba-7931f806abca" (UID: "fa6e64e5-fe59-4338-bbba-7931f806abca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:31.951268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.951186    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:31.951268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.951220    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kxfs\" (UniqueName: \"kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-kube-api-access-9kxfs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:31.951268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.951235    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa6e64e5-fe59-4338-bbba-7931f806abca-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:31.951268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:31.951247    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e64e5-fe59-4338-bbba-7931f806abca-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:32.178279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:32.178247    4918 generic.go:356] "Generic (PLEG): container finished" podID="fa6e64e5-fe59-4338-bbba-7931f806abca" containerID="d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a" exitCode=0
Oct 30 21:41:32.178502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:32.178305    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"
Oct 30 21:41:32.178502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:32.178340    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr" event={"ID":"fa6e64e5-fe59-4338-bbba-7931f806abca","Type":"ContainerDied","Data":"d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a"}
Oct 30 21:41:32.178502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:32.178406    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr" event={"ID":"fa6e64e5-fe59-4338-bbba-7931f806abca","Type":"ContainerDied","Data":"3eb4881d32919e8fb8f2408ac3313c83d866d1a57f4b4a8e04412c30eeb164e9"}
Oct 30 21:41:32.178502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:32.178436    4918 scope.go:122] "RemoveContainer" containerID="d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a"
Oct 30 21:41:32.191617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:32.191600    4918 scope.go:122] "RemoveContainer" containerID="d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a"
Oct 30 21:41:32.191876 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:32.191856    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a\": container with ID starting with d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a not found: ID does not exist" containerID="d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a"
Oct 30 21:41:32.191949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:32.191888    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a"} err="failed to get container status \"d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a\": rpc error: code = NotFound desc = could not find container \"d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a\": container with ID starting with d889a5c523fd5012180d1bfbe3d9be3fd528b45440440ff48937e3f77897575a not found: ID does not exist"
Oct 30 21:41:32.205776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:32.205697    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"]
Oct 30 21:41:32.211063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:32.211043    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-f69bgkcpr"]
Oct 30 21:41:32.690417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:32.689969    4918 scope.go:122] "RemoveContainer" containerID="d11ded123281b7af2e7764ac92c4831ce21b036ecf5793d3c85e4eac9c86e14f"
Oct 30 21:41:33.101568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:33.101534    4918 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" secret="" err="secret \"payload-processing-external-model-e2e--74dc6735-dockercfg-vqqdw\" not found"
Oct 30 21:41:33.213483 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:33.213455    4918 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-plugins-e2e-extmod-a-8072a0: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8072a0" not found
Oct 30 21:41:33.214918 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:33.214897    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume podName:37f91907-7b5d-4a7a-9912-5d9a121c7be1 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:33.714873479 +0000 UTC m=+2406.761432607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" (UID: "37f91907-7b5d-4a7a-9912-5d9a121c7be1") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8072a0" not found
Oct 30 21:41:33.216513 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:33.213515    4918 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8072a0/agc-cred-4099299197dc: secret "agc-cred-4099299197dc" not found
Oct 30 21:41:33.216651 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:33.216639    4918 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp: secret "agc-cred-4099299197dc" not found
Oct 30 21:41:33.216793 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:33.216780    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials podName:37f91907-7b5d-4a7a-9912-5d9a121c7be1 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:33.71676496 +0000 UTC m=+2406.763324084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" (UID: "37f91907-7b5d-4a7a-9912-5d9a121c7be1") : secret "agc-cred-4099299197dc" not found
Oct 30 21:41:33.231622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:33.231585    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa6e64e5-fe59-4338-bbba-7931f806abca" path="/var/lib/kubelet/pods/fa6e64e5-fe59-4338-bbba-7931f806abca/volumes"
Oct 30 21:41:33.232208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:33.232163    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"]
Oct 30 21:41:33.736427 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:33.734866    4918 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8072a0/agc-cred-4099299197dc: secret "agc-cred-4099299197dc" not found
Oct 30 21:41:33.736427 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:33.734895    4918 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp: secret "agc-cred-4099299197dc" not found
Oct 30 21:41:33.736427 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:33.734973    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials podName:37f91907-7b5d-4a7a-9912-5d9a121c7be1 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:34.734952291 +0000 UTC m=+2407.781511432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" (UID: "37f91907-7b5d-4a7a-9912-5d9a121c7be1") : secret "agc-cred-4099299197dc" not found
Oct 30 21:41:33.736427 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:33.735367    4918 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-plugins-e2e-extmod-a-8072a0: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8072a0" not found
Oct 30 21:41:33.736427 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:33.736109    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume podName:37f91907-7b5d-4a7a-9912-5d9a121c7be1 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:34.736089684 +0000 UTC m=+2407.782648830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" (UID: "37f91907-7b5d-4a7a-9912-5d9a121c7be1") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8072a0" not found
Oct 30 21:41:34.757857 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:34.757827    4918 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-plugins-e2e-extmod-a-8072a0: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8072a0" not found
Oct 30 21:41:34.758424 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:34.757889    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume podName:37f91907-7b5d-4a7a-9912-5d9a121c7be1 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:36.757875316 +0000 UTC m=+2409.804434440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" (UID: "37f91907-7b5d-4a7a-9912-5d9a121c7be1") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8072a0" not found
Oct 30 21:41:34.758424 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:34.757950    4918 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8072a0/agc-cred-4099299197dc: secret "agc-cred-4099299197dc" not found
Oct 30 21:41:34.758424 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:34.757971    4918 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp: secret "agc-cred-4099299197dc" not found
Oct 30 21:41:34.758424 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:34.758021    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials podName:37f91907-7b5d-4a7a-9912-5d9a121c7be1 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:36.758008947 +0000 UTC m=+2409.804568070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" (UID: "37f91907-7b5d-4a7a-9912-5d9a121c7be1") : secret "agc-cred-4099299197dc" not found
Oct 30 21:41:36.830033 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:36.829996    4918 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-plugins-e2e-extmod-a-8072a0: configmap "payload-processing-external-model-plugins-e2e-extmod-a-8072a0" not found
Oct 30 21:41:36.830584 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:36.830078    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume podName:37f91907-7b5d-4a7a-9912-5d9a121c7be1 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:40.830059045 +0000 UTC m=+2413.876618184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" (UID: "37f91907-7b5d-4a7a-9912-5d9a121c7be1") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-8072a0" not found
Oct 30 21:41:36.830584 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:36.830152    4918 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-8072a0/agc-cred-4099299197dc: secret "agc-cred-4099299197dc" not found
Oct 30 21:41:36.830584 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:36.830168    4918 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp: secret "agc-cred-4099299197dc" not found
Oct 30 21:41:36.830584 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:36.830224    4918 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials podName:37f91907-7b5d-4a7a-9912-5d9a121c7be1 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:40.830209301 +0000 UTC m=+2413.876768441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" (UID: "37f91907-7b5d-4a7a-9912-5d9a121c7be1") : secret "agc-cred-4099299197dc" not found
Oct 30 21:41:38.234797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.234736    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" podUID="37f91907-7b5d-4a7a-9912-5d9a121c7be1" containerName="payload-processing" containerID="cri-o://1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b" gracePeriod=25
Oct 30 21:41:38.305299 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:41:38.305276    4918 logging.go:55] [core] [Channel #8175 SubChannel #8176]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.90:50052", ServerName: "10.132.1.90:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.90:50052: connect: connection refused"
Oct 30 21:41:38.609584 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.609559    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:41:38.705010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.704975    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-kube-api-access-kk4p5\" (UniqueName: \"kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-kube-api-access-kk4p5\") pod \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") "
Oct 30 21:41:38.705193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.705026    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume\") pod \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") "
Oct 30 21:41:38.705363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.705341    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials\") pod \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") "
Oct 30 21:41:38.705498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.705383    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-routing-overlay-volume\") pod \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\" (UID: \"37f91907-7b5d-4a7a-9912-5d9a121c7be1\") "
Oct 30 21:41:38.705570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.705544    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume" pod "37f91907-7b5d-4a7a-9912-5d9a121c7be1" (UID: "37f91907-7b5d-4a7a-9912-5d9a121c7be1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:38.705815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.705782    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-routing-overlay-volume" pod "37f91907-7b5d-4a7a-9912-5d9a121c7be1" (UID: "37f91907-7b5d-4a7a-9912-5d9a121c7be1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:38.706244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.706226    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:38.706327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.706245    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f91907-7b5d-4a7a-9912-5d9a121c7be1-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:38.707683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.707660    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-kube-api-access-kk4p5" pod "37f91907-7b5d-4a7a-9912-5d9a121c7be1" (UID: "37f91907-7b5d-4a7a-9912-5d9a121c7be1"). InnerVolumeSpecName "kube-api-access-kk4p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:38.707820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.707804    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials" pod "37f91907-7b5d-4a7a-9912-5d9a121c7be1" (UID: "37f91907-7b5d-4a7a-9912-5d9a121c7be1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:38.807596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.807521    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk4p5\" (UniqueName: \"kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-kube-api-access-kk4p5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:38.807596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:38.807549    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/37f91907-7b5d-4a7a-9912-5d9a121c7be1-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:39.289290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:39.289255    4918 generic.go:356] "Generic (PLEG): container finished" podID="37f91907-7b5d-4a7a-9912-5d9a121c7be1" containerID="1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b" exitCode=0
Oct 30 21:41:39.289851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:39.289314    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"
Oct 30 21:41:39.289851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:39.289319    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" event={"ID":"37f91907-7b5d-4a7a-9912-5d9a121c7be1","Type":"ContainerDied","Data":"1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b"}
Oct 30 21:41:39.289851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:39.289466    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" event={"ID":"37f91907-7b5d-4a7a-9912-5d9a121c7be1","Type":"ContainerDied","Data":"e4cefbbfaad6aff3020848ddd7b5cf5abbd578c1a0de2532029804c9c9002b4a"}
Oct 30 21:41:39.289851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:39.289490    4918 scope.go:122] "RemoveContainer" containerID="1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b"
Oct 30 21:41:39.331578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:39.301918    4918 scope.go:122] "RemoveContainer" containerID="1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b"
Oct 30 21:41:39.331578 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:39.302243    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b\": container with ID starting with 1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b not found: ID does not exist" containerID="1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b"
Oct 30 21:41:39.331578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:39.302278    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b"} err="failed to get container status \"1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b\": rpc error: code = NotFound desc = could not find container \"1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b\": container with ID starting with 1f1656a96b11a412b37799a434c194eebfca54632cd0fdd54e826dd9034cf58b not found: ID does not exist"
Oct 30 21:41:39.373074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:39.372536    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp" podUID="37f91907-7b5d-4a7a-9912-5d9a121c7be1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.90:50052\" within 1s: context deadline exceeded"
Oct 30 21:41:39.373074 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:41:39.372962    4918 logging.go:55] [core] [Channel #8175 SubChannel #8176]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.90:50052", ServerName: "10.132.1.90:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.90:50052: operation was canceled"
Oct 30 21:41:39.389618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:39.389571    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"]
Oct 30 21:41:39.419177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:39.417246    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-69f4xsdlp"]
Oct 30 21:41:41.182417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:41.170660    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f91907-7b5d-4a7a-9912-5d9a121c7be1" path="/var/lib/kubelet/pods/37f91907-7b5d-4a7a-9912-5d9a121c7be1/volumes"
Oct 30 21:41:50.141065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:50.137508    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"]
Oct 30 21:41:50.146945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:50.146916    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-120178ce-xxd72"]
Oct 30 21:41:51.082337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:41:51.082299    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f2f1b1-e81c-4ed1-a74e-79477e0551ea" path="/var/lib/kubelet/pods/64f2f1b1-e81c-4ed1-a74e-79477e0551ea/volumes"
Oct 30 21:41:59.202853 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:41:59.202808    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:42:02.026116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:02.026086    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"]
Oct 30 21:42:02.031338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:02.031312    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe9de59-7wkrs"]
Oct 30 21:42:03.092144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:03.092089    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0b785f-07ed-4382-af03-5709574a77bd" path="/var/lib/kubelet/pods/fa0b785f-07ed-4382-af03-5709574a77bd/volumes"
Oct 30 21:42:15.029441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:15.029408    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"]
Oct 30 21:42:15.034631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:15.034606    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-080a7479-9bhvs"]
Oct 30 21:42:15.157678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:15.157644    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f17193-1e5c-42b9-b2d9-d959c80b2866" path="/var/lib/kubelet/pods/e7f17193-1e5c-42b9-b2d9-d959c80b2866/volumes"
Oct 30 21:42:26.027696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:26.027655    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"]
Oct 30 21:42:26.034869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:26.034835    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9278f1d5-smsbm"]
Oct 30 21:42:27.085518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:27.085490    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2794ef91-9efc-48fd-9baa-8aca4355fe64" path="/var/lib/kubelet/pods/2794ef91-9efc-48fd-9baa-8aca4355fe64/volumes"
Oct 30 21:42:29.198050 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:42:29.198008    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:42:50.865292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:50.865208    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"]
Oct 30 21:42:50.865838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:50.865521    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2" podUID="80e8c87e-a8e0-4051-9459-818dbf9e00e4" containerName="istio-proxy" containerID="cri-o://55a4da94b2829be0cb124af69f5815799b11bdd23170a893dc1e63b108798155" gracePeriod=30
Oct 30 21:42:52.039663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:52.039628    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"]
Oct 30 21:42:52.063342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:52.063307    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-531f9d9e-7rzsh"]
Oct 30 21:42:53.081126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:53.081097    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fda6676-8f64-426f-b41c-5f3abd8f240a" path="/var/lib/kubelet/pods/6fda6676-8f64-426f-b41c-5f3abd8f240a/volumes"
Oct 30 21:42:56.245109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.245062    4918 generic.go:356] "Generic (PLEG): container finished" podID="80e8c87e-a8e0-4051-9459-818dbf9e00e4" containerID="55a4da94b2829be0cb124af69f5815799b11bdd23170a893dc1e63b108798155" exitCode=0
Oct 30 21:42:56.245600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.245131    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2" event={"ID":"80e8c87e-a8e0-4051-9459-818dbf9e00e4","Type":"ContainerDied","Data":"55a4da94b2829be0cb124af69f5815799b11bdd23170a893dc1e63b108798155"}
Oct 30 21:42:56.289997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.289973    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:42:56.385907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.385870    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-credential-socket\") pod \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") "
Oct 30 21:42:56.386143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.386034    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-envoy\") pod \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") "
Oct 30 21:42:56.386143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.386117    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-kube-api-access-d58qg\" (UniqueName: \"kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-kube-api-access-d58qg\") pod \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") "
Oct 30 21:42:56.386292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.386180    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-credential-socket" pod "80e8c87e-a8e0-4051-9459-818dbf9e00e4" (UID: "80e8c87e-a8e0-4051-9459-818dbf9e00e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:56.386352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.386304    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-data\") pod \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") "
Oct 30 21:42:56.386438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.386350    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-token\" (UniqueName: \"kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-token\") pod \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") "
Oct 30 21:42:56.386509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.386470    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-socket\") pod \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") "
Oct 30 21:42:56.386567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.386513    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-podinfo\") pod \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") "
Oct 30 21:42:56.386625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.386595    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istiod-ca-cert\") pod \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") "
Oct 30 21:42:56.386800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.386777    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-certs\") pod \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\" (UID: \"80e8c87e-a8e0-4051-9459-818dbf9e00e4\") "
Oct 30 21:42:56.386885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.386828    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-socket" pod "80e8c87e-a8e0-4051-9459-818dbf9e00e4" (UID: "80e8c87e-a8e0-4051-9459-818dbf9e00e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:56.387099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.387067    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-data" pod "80e8c87e-a8e0-4051-9459-818dbf9e00e4" (UID: "80e8c87e-a8e0-4051-9459-818dbf9e00e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:56.388023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.387231    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istiod-ca-cert" pod "80e8c87e-a8e0-4051-9459-818dbf9e00e4" (UID: "80e8c87e-a8e0-4051-9459-818dbf9e00e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:56.388023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.387938    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:56.388023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.387964    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:56.388023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.387981    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:56.388023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.387996    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:56.388562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.388507    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-certs" pod "80e8c87e-a8e0-4051-9459-818dbf9e00e4" (UID: "80e8c87e-a8e0-4051-9459-818dbf9e00e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:56.389633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.389599    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-podinfo" pod "80e8c87e-a8e0-4051-9459-818dbf9e00e4" (UID: "80e8c87e-a8e0-4051-9459-818dbf9e00e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:42:56.389633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.389615    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-kube-api-access-d58qg" pod "80e8c87e-a8e0-4051-9459-818dbf9e00e4" (UID: "80e8c87e-a8e0-4051-9459-818dbf9e00e4"). InnerVolumeSpecName "kube-api-access-d58qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:56.389847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.389686    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-envoy" pod "80e8c87e-a8e0-4051-9459-818dbf9e00e4" (UID: "80e8c87e-a8e0-4051-9459-818dbf9e00e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:56.389988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.389964    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-token" pod "80e8c87e-a8e0-4051-9459-818dbf9e00e4" (UID: "80e8c87e-a8e0-4051-9459-818dbf9e00e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:56.490110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.490075    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:56.490110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.490107    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:56.490325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.490118    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58qg\" (UniqueName: \"kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-kube-api-access-d58qg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:56.490325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.490129    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:56.490325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:56.490138    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80e8c87e-a8e0-4051-9459-818dbf9e00e4-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:57.258427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:57.258381    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"
Oct 30 21:42:57.258427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:57.258413    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2" event={"ID":"80e8c87e-a8e0-4051-9459-818dbf9e00e4","Type":"ContainerDied","Data":"9befa53c4ccfac405645f98dfface4c0dd28a10f2083e243dcd4b9b93555fc4f"}
Oct 30 21:42:57.258996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:57.258490    4918 scope.go:122] "RemoveContainer" containerID="55a4da94b2829be0cb124af69f5815799b11bdd23170a893dc1e63b108798155"
Oct 30 21:42:57.292609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:57.292575    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"]
Oct 30 21:42:57.299860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:57.299830    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-dbcq2"]
Oct 30 21:42:59.134608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:42:59.134570    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e8c87e-a8e0-4051-9459-818dbf9e00e4" path="/var/lib/kubelet/pods/80e8c87e-a8e0-4051-9459-818dbf9e00e4/volumes"
Oct 30 21:42:59.164243 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:42:59.164201    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:43:03.052419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:03.051451    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"]
Oct 30 21:43:03.060456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:03.060415    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937adfda-dghmt"]
Oct 30 21:43:03.119369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:03.119338    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b9716a-4f3f-4c1d-8366-58967d60cddd" path="/var/lib/kubelet/pods/29b9716a-4f3f-4c1d-8366-58967d60cddd/volumes"
Oct 30 21:43:13.027846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:13.027810    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"]
Oct 30 21:43:13.032759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:13.032735    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c189cba-g7gj7"]
Oct 30 21:43:13.080426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:13.080376    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d04f9e5-4726-443a-bf43-674986770835" path="/var/lib/kubelet/pods/7d04f9e5-4726-443a-bf43-674986770835/volumes"
Oct 30 21:43:27.068151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:27.068109    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"]
Oct 30 21:43:27.077987 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:43:27.077954    4918 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-31445c7e-gvs94\" is forbidden: User \"system:node:ip-10-0-139-90.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-139-90.us-east-2.compute.internal' and this object" podUID="4789b259-18ec-40bc-a923-41f73bbc7d43" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"
Oct 30 21:43:27.081869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:27.081843    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31445c7e-gvs94"]
Oct 30 21:43:29.124670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:29.124633    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4789b259-18ec-40bc-a923-41f73bbc7d43" path="/var/lib/kubelet/pods/4789b259-18ec-40bc-a923-41f73bbc7d43/volumes"
Oct 30 21:43:29.170237 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:43:29.170194    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:43:33.127782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:33.127750    4918 scope.go:122] "RemoveContainer" containerID="32c9989db5ba1aac054172177985aa4035913740da565a84a750533bc21a2cbd"
Oct 30 21:43:33.139041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:33.139022    4918 scope.go:122] "RemoveContainer" containerID="0b7c5a59d93e225188d4b9320d2bc2e2a4af062f60ad2546c9420d820a45246b"
Oct 30 21:43:33.149090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:33.149065    4918 scope.go:122] "RemoveContainer" containerID="1e15d4a6a5eb0155f7d61622394fa8173aca0c94789223b6efbaff0f53db8c41"
Oct 30 21:43:33.160608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:33.160586    4918 scope.go:122] "RemoveContainer" containerID="5acd709be3abcd780e214054ce7a1408a548a8a3bdef0bad6505b356941ed812"
Oct 30 21:43:37.037516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:37.037482    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"]
Oct 30 21:43:37.044325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:37.044294    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4f333b4-kgp57"]
Oct 30 21:43:37.084016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:37.083973    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31454eb8-a101-4a6e-aedc-6e8d5588a827" path="/var/lib/kubelet/pods/31454eb8-a101-4a6e-aedc-6e8d5588a827/volumes"
Oct 30 21:43:48.047305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:48.047273    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"]
Oct 30 21:43:48.062034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:48.062006    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c9f1e32-6ksdg"]
Oct 30 21:43:49.094986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:43:49.094946    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ecda13-44be-4b82-9719-04ef06924244" path="/var/lib/kubelet/pods/08ecda13-44be-4b82-9719-04ef06924244/volumes"
Oct 30 21:43:59.161431 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:43:59.161350    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:44:02.039371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:02.039342    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"]
Oct 30 21:44:02.044330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:02.044299    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59d8258-cfrl4"]
Oct 30 21:44:03.081782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:03.081746    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03af4727-db4d-40d4-bbc5-8e13561cdf14" path="/var/lib/kubelet/pods/03af4727-db4d-40d4-bbc5-8e13561cdf14/volumes"
Oct 30 21:44:05.042788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:05.042757    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"]
Oct 30 21:44:05.048006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:05.047981    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d561d836-b7lxc"]
Oct 30 21:44:05.081283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:05.081257    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99d75b1-b597-4f77-99d1-1b3ef2f799a5" path="/var/lib/kubelet/pods/a99d75b1-b597-4f77-99d1-1b3ef2f799a5/volumes"
Oct 30 21:44:14.208566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.208488    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"]
Oct 30 21:44:14.209406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.209374    4918 cpu_manager.go:414] "containerMap: removing container" podUID="3fd993b1-2129-44db-9269-2943acd440b7" containerName="revoke-keys"
Oct 30 21:44:14.209458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.209410    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd993b1-2129-44db-9269-2943acd440b7" containerName="revoke-keys"
Oct 30 21:44:14.209458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.209424    4918 cpu_manager.go:414] "containerMap: removing container" podUID="37f91907-7b5d-4a7a-9912-5d9a121c7be1" containerName="payload-processing"
Oct 30 21:44:14.209458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.209429    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f91907-7b5d-4a7a-9912-5d9a121c7be1" containerName="payload-processing"
Oct 30 21:44:14.209458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.209457    4918 cpu_manager.go:414] "containerMap: removing container" podUID="80e8c87e-a8e0-4051-9459-818dbf9e00e4" containerName="istio-proxy"
Oct 30 21:44:14.209604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.209463    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e8c87e-a8e0-4051-9459-818dbf9e00e4" containerName="istio-proxy"
Oct 30 21:44:14.209604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.209488    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fa6e64e5-fe59-4338-bbba-7931f806abca" containerName="payload-processing"
Oct 30 21:44:14.209604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.209493    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6e64e5-fe59-4338-bbba-7931f806abca" containerName="payload-processing"
Oct 30 21:44:14.209703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.209628    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6e64e5-fe59-4338-bbba-7931f806abca" containerName="payload-processing"
Oct 30 21:44:14.209703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.209642    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd993b1-2129-44db-9269-2943acd440b7" containerName="revoke-keys"
Oct 30 21:44:14.209703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.209648    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f91907-7b5d-4a7a-9912-5d9a121c7be1" containerName="payload-processing"
Oct 30 21:44:14.209703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.209659    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e8c87e-a8e0-4051-9459-818dbf9e00e4" containerName="istio-proxy"
Oct 30 21:44:14.215049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.215031    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:14.221767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.221586    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:44:14.221767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.221737    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:44:14.223125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.223103    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:44:14.223292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.223277    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:44:14.229050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.229023    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"]
Oct 30 21:44:14.292440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.292380    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72cedf6b-bcszs\" (UID: \"e25da316-885a-432d-b00c-fa70d628c57c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:14.292623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.292597    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69jx6\" (UniqueName: \"kubernetes.io/projected/e25da316-885a-432d-b00c-fa70d628c57c-kube-api-access-69jx6\") pod \"maas-api-revoke-sub-72cedf6b-bcszs\" (UID: \"e25da316-885a-432d-b00c-fa70d628c57c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:14.292758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.292738    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72cedf6b-bcszs\" (UID: \"e25da316-885a-432d-b00c-fa70d628c57c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:14.393890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.393851    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72cedf6b-bcszs\" (UID: \"e25da316-885a-432d-b00c-fa70d628c57c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:14.394084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.393995    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69jx6\" (UniqueName: \"kubernetes.io/projected/e25da316-885a-432d-b00c-fa70d628c57c-kube-api-access-69jx6\") pod \"maas-api-revoke-sub-72cedf6b-bcszs\" (UID: \"e25da316-885a-432d-b00c-fa70d628c57c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:14.394163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.394108    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72cedf6b-bcszs\" (UID: \"e25da316-885a-432d-b00c-fa70d628c57c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:14.394766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.394743    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72cedf6b-bcszs\" (UID: \"e25da316-885a-432d-b00c-fa70d628c57c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:14.396603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.396572    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72cedf6b-bcszs\" (UID: \"e25da316-885a-432d-b00c-fa70d628c57c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:14.404063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.404040    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69jx6\" (UniqueName: \"kubernetes.io/projected/e25da316-885a-432d-b00c-fa70d628c57c-kube-api-access-69jx6\") pod \"maas-api-revoke-sub-72cedf6b-bcszs\" (UID: \"e25da316-885a-432d-b00c-fa70d628c57c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:14.533560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.533532    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:14.825558 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:44:14.825526    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25da316_885a_432d_b00c_fa70d628c57c.slice/crio-d30f62f9c7d7c59474dcd63f74991b5ee0abf0a86e7d4c070fbaa96d67b57e54 WatchSource:0}: Error finding container d30f62f9c7d7c59474dcd63f74991b5ee0abf0a86e7d4c070fbaa96d67b57e54: Status 404 returned error can't find the container with id d30f62f9c7d7c59474dcd63f74991b5ee0abf0a86e7d4c070fbaa96d67b57e54
Oct 30 21:44:14.827203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:14.827126    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"]
Oct 30 21:44:15.455905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:15.455820    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs" event={"ID":"e25da316-885a-432d-b00c-fa70d628c57c","Type":"ContainerStarted","Data":"eebd3a3b9cef64b769e8ff0bf87364da442478227c31fd8940dfd46033b900e1"}
Oct 30 21:44:15.455905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:15.455895    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs" event={"ID":"e25da316-885a-432d-b00c-fa70d628c57c","Type":"ContainerStarted","Data":"d30f62f9c7d7c59474dcd63f74991b5ee0abf0a86e7d4c070fbaa96d67b57e54"}
Oct 30 21:44:15.500098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:15.500051    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs" podStartSLOduration=1.500038851 podStartE2EDuration="1.500038851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:15.496348671 +0000 UTC m=+2568.542907828" watchObservedRunningTime="2026-10-30 21:44:15.500038851 +0000 UTC m=+2568.546598033"
Oct 30 21:44:16.466226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:16.466194    4918 generic.go:356] "Generic (PLEG): container finished" podID="e25da316-885a-432d-b00c-fa70d628c57c" containerID="eebd3a3b9cef64b769e8ff0bf87364da442478227c31fd8940dfd46033b900e1" exitCode=0
Oct 30 21:44:16.466691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:16.466286    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs" event={"ID":"e25da316-885a-432d-b00c-fa70d628c57c","Type":"ContainerDied","Data":"eebd3a3b9cef64b769e8ff0bf87364da442478227c31fd8940dfd46033b900e1"}
Oct 30 21:44:17.652564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:17.652536    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:17.765683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:17.765597    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25da316-885a-432d-b00c-fa70d628c57c-kube-api-access-69jx6\" (UniqueName: \"kubernetes.io/projected/e25da316-885a-432d-b00c-fa70d628c57c-kube-api-access-69jx6\") pod \"e25da316-885a-432d-b00c-fa70d628c57c\" (UID: \"e25da316-885a-432d-b00c-fa70d628c57c\") "
Oct 30 21:44:17.765863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:17.765756    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-openshift-ca\") pod \"e25da316-885a-432d-b00c-fa70d628c57c\" (UID: \"e25da316-885a-432d-b00c-fa70d628c57c\") "
Oct 30 21:44:17.766041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:17.766020    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-xks-ca\") pod \"e25da316-885a-432d-b00c-fa70d628c57c\" (UID: \"e25da316-885a-432d-b00c-fa70d628c57c\") "
Oct 30 21:44:17.766172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:17.766145    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-openshift-ca" pod "e25da316-885a-432d-b00c-fa70d628c57c" (UID: "e25da316-885a-432d-b00c-fa70d628c57c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:17.766650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:17.766624    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:17.785121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:17.768818    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-xks-ca" pod "e25da316-885a-432d-b00c-fa70d628c57c" (UID: "e25da316-885a-432d-b00c-fa70d628c57c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:17.785121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:17.784993    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25da316-885a-432d-b00c-fa70d628c57c-kube-api-access-69jx6" pod "e25da316-885a-432d-b00c-fa70d628c57c" (UID: "e25da316-885a-432d-b00c-fa70d628c57c"). InnerVolumeSpecName "kube-api-access-69jx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:17.871365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:17.871328    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e25da316-885a-432d-b00c-fa70d628c57c-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:17.871365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:17.871357    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69jx6\" (UniqueName: \"kubernetes.io/projected/e25da316-885a-432d-b00c-fa70d628c57c-kube-api-access-69jx6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:18.484743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:18.484715    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"
Oct 30 21:44:18.484932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:18.484763    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs" event={"ID":"e25da316-885a-432d-b00c-fa70d628c57c","Type":"ContainerDied","Data":"d30f62f9c7d7c59474dcd63f74991b5ee0abf0a86e7d4c070fbaa96d67b57e54"}
Oct 30 21:44:18.484932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:18.484820    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d30f62f9c7d7c59474dcd63f74991b5ee0abf0a86e7d4c070fbaa96d67b57e54"
Oct 30 21:44:26.054406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.054361    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"]
Oct 30 21:44:26.055241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.055225    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e25da316-885a-432d-b00c-fa70d628c57c" containerName="revoke-keys"
Oct 30 21:44:26.055287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.055244    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25da316-885a-432d-b00c-fa70d628c57c" containerName="revoke-keys"
Oct 30 21:44:26.055424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.055381    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e25da316-885a-432d-b00c-fa70d628c57c" containerName="revoke-keys"
Oct 30 21:44:26.059293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.059277    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:26.067430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.066199    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:44:26.067430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.066258    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:44:26.067430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.067262    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:44:26.067666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.067486    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:44:26.071135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.071111    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"]
Oct 30 21:44:26.133167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.133131    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0da6cadf-dhgf5\" (UID: \"745e7408-a27f-482f-ba79-8d8741c0ef7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:26.133494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.133470    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn26l\" (UniqueName: \"kubernetes.io/projected/745e7408-a27f-482f-ba79-8d8741c0ef7d-kube-api-access-fn26l\") pod \"maas-api-revoke-sub-0da6cadf-dhgf5\" (UID: \"745e7408-a27f-482f-ba79-8d8741c0ef7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:26.133621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.133605    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0da6cadf-dhgf5\" (UID: \"745e7408-a27f-482f-ba79-8d8741c0ef7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:26.234992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.234962    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn26l\" (UniqueName: \"kubernetes.io/projected/745e7408-a27f-482f-ba79-8d8741c0ef7d-kube-api-access-fn26l\") pod \"maas-api-revoke-sub-0da6cadf-dhgf5\" (UID: \"745e7408-a27f-482f-ba79-8d8741c0ef7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:26.235172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.235053    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0da6cadf-dhgf5\" (UID: \"745e7408-a27f-482f-ba79-8d8741c0ef7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:26.235255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.235179    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0da6cadf-dhgf5\" (UID: \"745e7408-a27f-482f-ba79-8d8741c0ef7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:26.235852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.235828    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0da6cadf-dhgf5\" (UID: \"745e7408-a27f-482f-ba79-8d8741c0ef7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:26.237580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.237557    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0da6cadf-dhgf5\" (UID: \"745e7408-a27f-482f-ba79-8d8741c0ef7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:26.245811 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.245533    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn26l\" (UniqueName: \"kubernetes.io/projected/745e7408-a27f-482f-ba79-8d8741c0ef7d-kube-api-access-fn26l\") pod \"maas-api-revoke-sub-0da6cadf-dhgf5\" (UID: \"745e7408-a27f-482f-ba79-8d8741c0ef7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:26.398855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.391241    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:26.741852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:26.741794    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"]
Oct 30 21:44:27.038994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:27.038905    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"]
Oct 30 21:44:27.043924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:27.043895    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc417a70-jszc5"]
Oct 30 21:44:27.080925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:27.080897    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6104801-7f62-47d0-ab21-efaf9cda4484" path="/var/lib/kubelet/pods/b6104801-7f62-47d0-ab21-efaf9cda4484/volumes"
Oct 30 21:44:27.590656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:27.590619    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5" event={"ID":"745e7408-a27f-482f-ba79-8d8741c0ef7d","Type":"ContainerStarted","Data":"8975553e65d2eb206068f1db3e675211f97ffa9674c04f9212eae997a99171d8"}
Oct 30 21:44:27.590840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:27.590680    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5" event={"ID":"745e7408-a27f-482f-ba79-8d8741c0ef7d","Type":"ContainerStarted","Data":"39375c4d2f263d2983cbc853fc1e75660eb43cce64b9da111bd499740637448e"}
Oct 30 21:44:27.621915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:27.621860    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5" podStartSLOduration=1.62184694 podStartE2EDuration="1.62184694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:27.615700754 +0000 UTC m=+2580.662259918" watchObservedRunningTime="2026-10-30 21:44:27.62184694 +0000 UTC m=+2580.668406084"
Oct 30 21:44:28.600798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:28.600767    4918 generic.go:356] "Generic (PLEG): container finished" podID="745e7408-a27f-482f-ba79-8d8741c0ef7d" containerID="8975553e65d2eb206068f1db3e675211f97ffa9674c04f9212eae997a99171d8" exitCode=0
Oct 30 21:44:28.601209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:28.600851    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5" event={"ID":"745e7408-a27f-482f-ba79-8d8741c0ef7d","Type":"ContainerDied","Data":"8975553e65d2eb206068f1db3e675211f97ffa9674c04f9212eae997a99171d8"}
Oct 30 21:44:29.339504 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:44:29.339462    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:44:29.809336 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:29.809308    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:29.865049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:29.864988    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-xks-ca\") pod \"745e7408-a27f-482f-ba79-8d8741c0ef7d\" (UID: \"745e7408-a27f-482f-ba79-8d8741c0ef7d\") "
Oct 30 21:44:29.865049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:29.865042    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745e7408-a27f-482f-ba79-8d8741c0ef7d-kube-api-access-fn26l\" (UniqueName: \"kubernetes.io/projected/745e7408-a27f-482f-ba79-8d8741c0ef7d-kube-api-access-fn26l\") pod \"745e7408-a27f-482f-ba79-8d8741c0ef7d\" (UID: \"745e7408-a27f-482f-ba79-8d8741c0ef7d\") "
Oct 30 21:44:29.865226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:29.865178    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-openshift-ca\") pod \"745e7408-a27f-482f-ba79-8d8741c0ef7d\" (UID: \"745e7408-a27f-482f-ba79-8d8741c0ef7d\") "
Oct 30 21:44:29.865539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:29.865515    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-openshift-ca" pod "745e7408-a27f-482f-ba79-8d8741c0ef7d" (UID: "745e7408-a27f-482f-ba79-8d8741c0ef7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:29.866011 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:29.865991    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:29.867661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:29.867632    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-xks-ca" pod "745e7408-a27f-482f-ba79-8d8741c0ef7d" (UID: "745e7408-a27f-482f-ba79-8d8741c0ef7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:29.867892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:29.867777    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745e7408-a27f-482f-ba79-8d8741c0ef7d-kube-api-access-fn26l" pod "745e7408-a27f-482f-ba79-8d8741c0ef7d" (UID: "745e7408-a27f-482f-ba79-8d8741c0ef7d"). InnerVolumeSpecName "kube-api-access-fn26l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:29.968143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:29.968104    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/745e7408-a27f-482f-ba79-8d8741c0ef7d-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:29.968143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:29.968137    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn26l\" (UniqueName: \"kubernetes.io/projected/745e7408-a27f-482f-ba79-8d8741c0ef7d-kube-api-access-fn26l\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:30.620239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:30.620211    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"
Oct 30 21:44:30.620612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:30.620251    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5" event={"ID":"745e7408-a27f-482f-ba79-8d8741c0ef7d","Type":"ContainerDied","Data":"39375c4d2f263d2983cbc853fc1e75660eb43cce64b9da111bd499740637448e"}
Oct 30 21:44:30.620612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:30.620341    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39375c4d2f263d2983cbc853fc1e75660eb43cce64b9da111bd499740637448e"
Oct 30 21:44:33.219324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:33.219287    4918 scope.go:122] "RemoveContainer" containerID="8a2fd7d5f8a486b745ca3b1b64ba5ac543407c79d64c8470ca9a8564f62b7f4e"
Oct 30 21:44:33.231283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:33.231262    4918 scope.go:122] "RemoveContainer" containerID="9fb516aec342c1b6d7f77a5c291fa299c21ac2820fc10b55b5f0a1d16f1f91b2"
Oct 30 21:44:33.242840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:33.242815    4918 scope.go:122] "RemoveContainer" containerID="c024b4c647e84c4101be529330a4009c6d6570b23cedf28ef1ce5a37e4468ca6"
Oct 30 21:44:33.254494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:33.254466    4918 scope.go:122] "RemoveContainer" containerID="f4b93013abf0fd76fead34cac8274b928e7f341bd8daa13b172bd7835584aae1"
Oct 30 21:44:34.047878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:34.047844    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"]
Oct 30 21:44:34.049190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:34.049128    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy" containerID="cri-o://4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c" gracePeriod=30
Oct 30 21:44:36.897701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:36.897668    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"]
Oct 30 21:44:36.898932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:36.898914    4918 cpu_manager.go:414] "containerMap: removing container" podUID="745e7408-a27f-482f-ba79-8d8741c0ef7d" containerName="revoke-keys"
Oct 30 21:44:36.898984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:36.898936    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745e7408-a27f-482f-ba79-8d8741c0ef7d" containerName="revoke-keys"
Oct 30 21:44:36.899169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:36.899158    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="745e7408-a27f-482f-ba79-8d8741c0ef7d" containerName="revoke-keys"
Oct 30 21:44:36.905249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:36.905228    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:36.912991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:36.912966    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:44:36.913247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:36.913226    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:44:36.913366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:36.913261    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:44:36.913366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:36.913259    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:44:36.931233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:36.931206    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"]
Oct 30 21:44:37.044634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:37.044577    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa74ed49-7x9xh\" (UID: \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:37.045187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:37.045164    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgvmw\" (UniqueName: \"kubernetes.io/projected/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-kube-api-access-jgvmw\") pod \"maas-api-revoke-sub-fa74ed49-7x9xh\" (UID: \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:37.045457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:37.045441    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa74ed49-7x9xh\" (UID: \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:37.147227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:37.147194    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgvmw\" (UniqueName: \"kubernetes.io/projected/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-kube-api-access-jgvmw\") pod \"maas-api-revoke-sub-fa74ed49-7x9xh\" (UID: \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:37.147472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:37.147298    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa74ed49-7x9xh\" (UID: \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:37.147632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:37.147613    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa74ed49-7x9xh\" (UID: \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:37.148156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:37.148110    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa74ed49-7x9xh\" (UID: \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:37.150149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:37.150129    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa74ed49-7x9xh\" (UID: \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:37.165992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:37.165968    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgvmw\" (UniqueName: \"kubernetes.io/projected/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-kube-api-access-jgvmw\") pod \"maas-api-revoke-sub-fa74ed49-7x9xh\" (UID: \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:37.220939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:37.220900    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:37.522090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:37.522056    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"]
Oct 30 21:44:37.524802 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:44:37.524773    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2eeeff_ccbe_41df_aaf5_5cabfc8965bb.slice/crio-2a67eb970e96c16eac05809dee75bc08c3d8e8a5116c2fb49d8fa72b8acf308a WatchSource:0}: Error finding container 2a67eb970e96c16eac05809dee75bc08c3d8e8a5116c2fb49d8fa72b8acf308a: Status 404 returned error can't find the container with id 2a67eb970e96c16eac05809dee75bc08c3d8e8a5116c2fb49d8fa72b8acf308a
Oct 30 21:44:38.010415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:38.009961    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh" event={"ID":"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb","Type":"ContainerStarted","Data":"2a67eb970e96c16eac05809dee75bc08c3d8e8a5116c2fb49d8fa72b8acf308a"}
Oct 30 21:44:38.097000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:38.096966    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"]
Oct 30 21:44:38.102722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:38.102696    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59086576-j2btd"]
Oct 30 21:44:39.020586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.020549    4918 generic.go:356] "Generic (PLEG): container finished" podID="9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb" containerID="91c1998096c198a8cf8674c0e243b76607b8d18f61ab2cdde1bacd884ac4e7cc" exitCode=0
Oct 30 21:44:39.021079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.020637    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh" event={"ID":"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb","Type":"ContainerDied","Data":"91c1998096c198a8cf8674c0e243b76607b8d18f61ab2cdde1bacd884ac4e7cc"}
Oct 30 21:44:39.029982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.029951    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"]
Oct 30 21:44:39.035380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.035356    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1db39592-pwtz9"]
Oct 30 21:44:39.183784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.183334    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51190d58-33e9-49de-b6ce-4086f1e48254" path="/var/lib/kubelet/pods/51190d58-33e9-49de-b6ce-4086f1e48254/volumes"
Oct 30 21:44:39.184230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.184194    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6def2c6d-d2fa-4f90-8dcc-4e2a2507e956" path="/var/lib/kubelet/pods/6def2c6d-d2fa-4f90-8dcc-4e2a2507e956/volumes"
Oct 30 21:44:39.403915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.403888    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:44:39.421145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421118    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-kube-api-access-qxds5\" (UniqueName: \"kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-kube-api-access-qxds5\") pod \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") "
Oct 30 21:44:39.421286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421259    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-socket\") pod \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") "
Oct 30 21:44:39.421349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421302    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-data\") pod \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") "
Oct 30 21:44:39.421349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421334    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-credential-socket\") pod \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") "
Oct 30 21:44:39.421477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421385    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istiod-ca-cert\") pod \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") "
Oct 30 21:44:39.421557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421541    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-podinfo\") pod \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") "
Oct 30 21:44:39.421624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421574    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-certs\") pod \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") "
Oct 30 21:44:39.421624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421591    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-socket" pod "f23c4448-ef53-4fe3-9b8d-6e69f5711f53" (UID: "f23c4448-ef53-4fe3-9b8d-6e69f5711f53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:39.421624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421608    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-data" pod "f23c4448-ef53-4fe3-9b8d-6e69f5711f53" (UID: "f23c4448-ef53-4fe3-9b8d-6e69f5711f53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:39.421798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421627    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-credential-socket" pod "f23c4448-ef53-4fe3-9b8d-6e69f5711f53" (UID: "f23c4448-ef53-4fe3-9b8d-6e69f5711f53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:39.421798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421655    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-envoy\") pod \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") "
Oct 30 21:44:39.421927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421830    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-certs" pod "f23c4448-ef53-4fe3-9b8d-6e69f5711f53" (UID: "f23c4448-ef53-4fe3-9b8d-6e69f5711f53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:39.421927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421833    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istiod-ca-cert" pod "f23c4448-ef53-4fe3-9b8d-6e69f5711f53" (UID: "f23c4448-ef53-4fe3-9b8d-6e69f5711f53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:39.421927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.421876    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-token\" (UniqueName: \"kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-token\") pod \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\" (UID: \"f23c4448-ef53-4fe3-9b8d-6e69f5711f53\") "
Oct 30 21:44:39.422806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.422778    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:39.422941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.422809    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:39.422941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.422824    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:39.422941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.422838    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:39.422941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.422852    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:39.423727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.423700    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-kube-api-access-qxds5" pod "f23c4448-ef53-4fe3-9b8d-6e69f5711f53" (UID: "f23c4448-ef53-4fe3-9b8d-6e69f5711f53"). InnerVolumeSpecName "kube-api-access-qxds5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:39.424058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.424031    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-envoy" pod "f23c4448-ef53-4fe3-9b8d-6e69f5711f53" (UID: "f23c4448-ef53-4fe3-9b8d-6e69f5711f53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:39.424171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.424154    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-token" pod "f23c4448-ef53-4fe3-9b8d-6e69f5711f53" (UID: "f23c4448-ef53-4fe3-9b8d-6e69f5711f53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:39.424234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.424182    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-podinfo" pod "f23c4448-ef53-4fe3-9b8d-6e69f5711f53" (UID: "f23c4448-ef53-4fe3-9b8d-6e69f5711f53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:44:39.524244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.524208    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:39.524244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.524241    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:39.524528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.524254    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:39.524528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:39.524267    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxds5\" (UniqueName: \"kubernetes.io/projected/f23c4448-ef53-4fe3-9b8d-6e69f5711f53-kube-api-access-qxds5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:40.031227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.031190    4918 generic.go:356] "Generic (PLEG): container finished" podID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerID="4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c" exitCode=0
Oct 30 21:44:40.031757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.031237    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"
Oct 30 21:44:40.031757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.031263    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" event={"ID":"f23c4448-ef53-4fe3-9b8d-6e69f5711f53","Type":"ContainerDied","Data":"4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c"}
Oct 30 21:44:40.031757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.031313    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk" event={"ID":"f23c4448-ef53-4fe3-9b8d-6e69f5711f53","Type":"ContainerDied","Data":"526af3da974c4f1081a80d742ade782ead96941c13bbc050348dd383dd64d599"}
Oct 30 21:44:40.031757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.031338    4918 scope.go:122] "RemoveContainer" containerID="4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c"
Oct 30 21:44:40.046085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.046067    4918 scope.go:122] "RemoveContainer" containerID="6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b"
Oct 30 21:44:40.061037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.060151    4918 scope.go:122] "RemoveContainer" containerID="4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c"
Oct 30 21:44:40.061037 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:44:40.060723    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c\": container with ID starting with 4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c not found: ID does not exist" containerID="4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c"
Oct 30 21:44:40.061037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.060765    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c"} err="failed to get container status \"4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c\": rpc error: code = NotFound desc = could not find container \"4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c\": container with ID starting with 4f8be9ad683feafb40933ca3e5a30cd2528c04191b4015d61aaee1dbe55e746c not found: ID does not exist"
Oct 30 21:44:40.061037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.060789    4918 scope.go:122] "RemoveContainer" containerID="6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b"
Oct 30 21:44:40.061608 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:44:40.061125    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b\": container with ID starting with 6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b not found: ID does not exist" containerID="6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b"
Oct 30 21:44:40.061608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.061158    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b"} err="failed to get container status \"6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b\": rpc error: code = NotFound desc = could not find container \"6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b\": container with ID starting with 6e28e98c4b2761bcfe65177a5968520c8015926434c28b6f664e16c1cbb7d96b not found: ID does not exist"
Oct 30 21:44:40.071550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.063963    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"]
Oct 30 21:44:40.074135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.074113    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-j97gk"]
Oct 30 21:44:40.264050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.264026    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:40.342222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.342130    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-openshift-ca\") pod \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\" (UID: \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\") "
Oct 30 21:44:40.342446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.342240    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-kube-api-access-jgvmw\" (UniqueName: \"kubernetes.io/projected/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-kube-api-access-jgvmw\") pod \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\" (UID: \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\") "
Oct 30 21:44:40.342567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.342516    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-xks-ca\") pod \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\" (UID: \"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb\") "
Oct 30 21:44:40.342655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.342621    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-openshift-ca" pod "9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb" (UID: "9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:40.343560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.343497    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:40.345004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.344974    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-xks-ca" pod "9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb" (UID: "9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:40.345319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.345301    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-kube-api-access-jgvmw" pod "9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb" (UID: "9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb"). InnerVolumeSpecName "kube-api-access-jgvmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:40.445032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.445003    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgvmw\" (UniqueName: \"kubernetes.io/projected/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-kube-api-access-jgvmw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:40.445032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:40.445031    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:41.040752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:41.040725    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"
Oct 30 21:44:41.040752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:41.040741    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh" event={"ID":"9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb","Type":"ContainerDied","Data":"2a67eb970e96c16eac05809dee75bc08c3d8e8a5116c2fb49d8fa72b8acf308a"}
Oct 30 21:44:41.041232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:41.040793    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a67eb970e96c16eac05809dee75bc08c3d8e8a5116c2fb49d8fa72b8acf308a"
Oct 30 21:44:41.080541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:41.080506    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" path="/var/lib/kubelet/pods/f23c4448-ef53-4fe3-9b8d-6e69f5711f53/volumes"
Oct 30 21:44:45.591842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:45.591801    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"]
Oct 30 21:44:45.592421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:45.592080    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" podUID="02880084-8170-487e-9fdf-0e9a3c7ba287" containerName="istio-proxy" containerID="cri-o://b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee" gracePeriod=30
Oct 30 21:44:50.068724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.068673    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"]
Oct 30 21:44:50.075476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.075449    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f20740d0-j2pff"]
Oct 30 21:44:50.188423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.188379    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"]
Oct 30 21:44:50.189212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.189198    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy"
Oct 30 21:44:50.189258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.189214    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy"
Oct 30 21:44:50.189258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.189230    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy"
Oct 30 21:44:50.189258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.189235    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy"
Oct 30 21:44:50.189362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.189283    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb" containerName="revoke-keys"
Oct 30 21:44:50.189362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.189288    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb" containerName="revoke-keys"
Oct 30 21:44:50.189468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.189426    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb" containerName="revoke-keys"
Oct 30 21:44:50.189468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.189440    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy"
Oct 30 21:44:50.189468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.189448    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23c4448-ef53-4fe3-9b8d-6e69f5711f53" containerName="istio-proxy"
Oct 30 21:44:50.195108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.195087    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.209958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.209931    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"]
Oct 30 21:44:50.222340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.222310    4918 patch_prober.go:29] interesting pod/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:44:50.222518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.222374    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" podUID="02880084-8170-487e-9fdf-0e9a3c7ba287" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:44:50.450468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.450275    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-provider-credentials\") pod \"payload-processing-external-model-5d847fc69b-fpqks\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") " pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.450468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.450343    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smdws\" (UniqueName: \"kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-kube-api-access-smdws\") pod \"payload-processing-external-model-5d847fc69b-fpqks\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") " pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.462427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.459663    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-plugins-config-volume\") pod \"payload-processing-external-model-5d847fc69b-fpqks\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") " pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.462427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.459804    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-routing-overlay-volume\") pod \"payload-processing-external-model-5d847fc69b-fpqks\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") " pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.598414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.570451    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-plugins-config-volume\") pod \"payload-processing-external-model-5d847fc69b-fpqks\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") " pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.598414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.570586    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-routing-overlay-volume\") pod \"payload-processing-external-model-5d847fc69b-fpqks\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") " pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.598414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.570793    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-provider-credentials\") pod \"payload-processing-external-model-5d847fc69b-fpqks\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") " pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.598414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.570845    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smdws\" (UniqueName: \"kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-kube-api-access-smdws\") pod \"payload-processing-external-model-5d847fc69b-fpqks\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") " pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.598414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.572214    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-routing-overlay-volume\") pod \"payload-processing-external-model-5d847fc69b-fpqks\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") " pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.598414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.572366    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-plugins-config-volume\") pod \"payload-processing-external-model-5d847fc69b-fpqks\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") " pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.616765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.616725    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-provider-credentials\") pod \"payload-processing-external-model-5d847fc69b-fpqks\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") " pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.639615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.633501    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smdws\" (UniqueName: \"kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-kube-api-access-smdws\") pod \"payload-processing-external-model-5d847fc69b-fpqks\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") " pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.814286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.812647    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:50.900540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.900509    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:44:50.982447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.982412    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-data\") pod \"02880084-8170-487e-9fdf-0e9a3c7ba287\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") "
Oct 30 21:44:50.982699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.982465    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-certs\") pod \"02880084-8170-487e-9fdf-0e9a3c7ba287\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") "
Oct 30 21:44:50.982699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.982521    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-token\" (UniqueName: \"kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-token\") pod \"02880084-8170-487e-9fdf-0e9a3c7ba287\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") "
Oct 30 21:44:50.982834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.982740    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-envoy\") pod \"02880084-8170-487e-9fdf-0e9a3c7ba287\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") "
Oct 30 21:44:50.982834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.982776    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-podinfo\") pod \"02880084-8170-487e-9fdf-0e9a3c7ba287\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") "
Oct 30 21:44:50.982939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.982849    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-socket\") pod \"02880084-8170-487e-9fdf-0e9a3c7ba287\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") "
Oct 30 21:44:50.982999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.982973    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-credential-socket\") pod \"02880084-8170-487e-9fdf-0e9a3c7ba287\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") "
Oct 30 21:44:50.983059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.983022    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02880084-8170-487e-9fdf-0e9a3c7ba287-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02880084-8170-487e-9fdf-0e9a3c7ba287-istiod-ca-cert\") pod \"02880084-8170-487e-9fdf-0e9a3c7ba287\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") "
Oct 30 21:44:50.983111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.983053    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-kube-api-access-94dg5\" (UniqueName: \"kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-kube-api-access-94dg5\") pod \"02880084-8170-487e-9fdf-0e9a3c7ba287\" (UID: \"02880084-8170-487e-9fdf-0e9a3c7ba287\") "
Oct 30 21:44:50.984961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.984927    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-certs" pod "02880084-8170-487e-9fdf-0e9a3c7ba287" (UID: "02880084-8170-487e-9fdf-0e9a3c7ba287"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:50.985163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.985140    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-data" pod "02880084-8170-487e-9fdf-0e9a3c7ba287" (UID: "02880084-8170-487e-9fdf-0e9a3c7ba287"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:50.985487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.985462    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02880084-8170-487e-9fdf-0e9a3c7ba287-istiod-ca-cert" pod "02880084-8170-487e-9fdf-0e9a3c7ba287" (UID: "02880084-8170-487e-9fdf-0e9a3c7ba287"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:50.986404 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.986359    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-credential-socket" pod "02880084-8170-487e-9fdf-0e9a3c7ba287" (UID: "02880084-8170-487e-9fdf-0e9a3c7ba287"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:50.986515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.986446    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-socket" pod "02880084-8170-487e-9fdf-0e9a3c7ba287" (UID: "02880084-8170-487e-9fdf-0e9a3c7ba287"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:50.989426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.989162    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-envoy" pod "02880084-8170-487e-9fdf-0e9a3c7ba287" (UID: "02880084-8170-487e-9fdf-0e9a3c7ba287"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:50.993521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.993484    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-podinfo" pod "02880084-8170-487e-9fdf-0e9a3c7ba287" (UID: "02880084-8170-487e-9fdf-0e9a3c7ba287"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:44:50.994582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.994555    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-token" pod "02880084-8170-487e-9fdf-0e9a3c7ba287" (UID: "02880084-8170-487e-9fdf-0e9a3c7ba287"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:50.996337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:50.996314    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-kube-api-access-94dg5" pod "02880084-8170-487e-9fdf-0e9a3c7ba287" (UID: "02880084-8170-487e-9fdf-0e9a3c7ba287"). InnerVolumeSpecName "kube-api-access-94dg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:51.081349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.081317    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51cb2578-9925-448f-b941-0fc2c8991373" path="/var/lib/kubelet/pods/51cb2578-9925-448f-b941-0fc2c8991373/volumes"
Oct 30 21:44:51.085697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.085670    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.085842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.085701    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.085842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.085717    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.085842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.085729    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.085842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.085740    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02880084-8170-487e-9fdf-0e9a3c7ba287-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.085842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.085748    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94dg5\" (UniqueName: \"kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-kube-api-access-94dg5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.085842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.085757    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.085842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.085767    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02880084-8170-487e-9fdf-0e9a3c7ba287-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.085842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.085775    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02880084-8170-487e-9fdf-0e9a3c7ba287-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:51.130005 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:44:51.129978    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc465785a_a77a_4721_b3a5_551b9ff2e394.slice/crio-62a1a023a41cbe8f6e150628d8a49bc7a34a5757854f7d1c572be86da5bcbd2f WatchSource:0}: Error finding container 62a1a023a41cbe8f6e150628d8a49bc7a34a5757854f7d1c572be86da5bcbd2f: Status 404 returned error can't find the container with id 62a1a023a41cbe8f6e150628d8a49bc7a34a5757854f7d1c572be86da5bcbd2f
Oct 30 21:44:51.132339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.132316    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"]
Oct 30 21:44:51.254563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.254460    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks" event={"ID":"c465785a-a77a-4721-b3a5-551b9ff2e394","Type":"ContainerStarted","Data":"2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee"}
Oct 30 21:44:51.255414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.255354    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks" event={"ID":"c465785a-a77a-4721-b3a5-551b9ff2e394","Type":"ContainerStarted","Data":"62a1a023a41cbe8f6e150628d8a49bc7a34a5757854f7d1c572be86da5bcbd2f"}
Oct 30 21:44:51.255566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.255422    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:44:51.256860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.256821    4918 generic.go:356] "Generic (PLEG): container finished" podID="02880084-8170-487e-9fdf-0e9a3c7ba287" containerID="b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee" exitCode=0
Oct 30 21:44:51.256995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.256981    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" event={"ID":"02880084-8170-487e-9fdf-0e9a3c7ba287","Type":"ContainerDied","Data":"b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee"}
Oct 30 21:44:51.257076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.257020    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n" event={"ID":"02880084-8170-487e-9fdf-0e9a3c7ba287","Type":"ContainerDied","Data":"b49d44d664ab2dbedae4fdd42bd1649104e7eabe969888f90e8effe2b569e7c4"}
Oct 30 21:44:51.257076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.257043    4918 scope.go:122] "RemoveContainer" containerID="b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee"
Oct 30 21:44:51.257195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.257179    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"
Oct 30 21:44:51.430475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.410525    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks" podStartSLOduration=1.4105042939999999 podStartE2EDuration="1.410504294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:51.326759426 +0000 UTC m=+2604.373318573" watchObservedRunningTime="2026-10-30 21:44:51.410504294 +0000 UTC m=+2604.457063442"
Oct 30 21:44:51.459418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.454831    4918 scope.go:122] "RemoveContainer" containerID="b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee"
Oct 30 21:44:51.475416 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:44:51.467586    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee\": container with ID starting with b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee not found: ID does not exist" containerID="b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee"
Oct 30 21:44:51.475416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.467648    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee"} err="failed to get container status \"b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee\": rpc error: code = NotFound desc = could not find container \"b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee\": container with ID starting with b4b2a595283e12a89ccbe2ac7e225b90a21dfb915d822a88edfff02eb52410ee not found: ID does not exist"
Oct 30 21:44:51.492420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.487866    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"]
Oct 30 21:44:51.582465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:51.579513    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-jf55n"]
Oct 30 21:44:52.192418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:52.184495    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"]
Oct 30 21:44:52.225412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:52.207906    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cce0ace-p54ws"]
Oct 30 21:44:52.663307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:52.662715    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:44:52.860475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:52.859567    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:44:52.895426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:52.887483    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:44:52.915491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:52.908721    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:44:52.932430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:52.927490    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:44:52.956427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:52.948499    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:44:52.980425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:52.977415    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:44:52.996574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:52.996502    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:44:53.045771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:53.045491    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:44:53.140435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:53.138482    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:44:53.219351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:53.219263    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02880084-8170-487e-9fdf-0e9a3c7ba287" path="/var/lib/kubelet/pods/02880084-8170-487e-9fdf-0e9a3c7ba287/volumes"
Oct 30 21:44:53.236361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:53.236242    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aabba317-ead4-4fac-ab08-a6f16eb9bd5b" path="/var/lib/kubelet/pods/aabba317-ead4-4fac-ab08-a6f16eb9bd5b/volumes"
Oct 30 21:44:53.664087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:53.664041    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:44:53.665233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:53.665171    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:44:53.776632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:53.776082    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:44:56.646334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.646299    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"]
Oct 30 21:44:56.647179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.647164    4918 cpu_manager.go:414] "containerMap: removing container" podUID="02880084-8170-487e-9fdf-0e9a3c7ba287" containerName="istio-proxy"
Oct 30 21:44:56.647239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.647180    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02880084-8170-487e-9fdf-0e9a3c7ba287" containerName="istio-proxy"
Oct 30 21:44:56.647321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.647311    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="02880084-8170-487e-9fdf-0e9a3c7ba287" containerName="istio-proxy"
Oct 30 21:44:56.651516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.651500    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:56.663219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.663156    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"agc-cred-4bb46bb46957\""
Oct 30 21:44:56.663752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.663730    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"openshift-service-ca.crt\""
Oct 30 21:44:56.663868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.663763    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e596df\""
Oct 30 21:44:56.663868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.663841    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"kube-root-ca.crt\""
Oct 30 21:44:56.664000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.663983    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"routing-overlay\""
Oct 30 21:44:56.665709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.665686    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"]
Oct 30 21:44:56.684295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.684270    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"agc-cred-0e714968df36\""
Oct 30 21:44:56.684506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.684481    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e596df\"/\"payload-processing-external-model-e2e--57c066ea-dockercfg-nkm9k\""
Oct 30 21:44:56.767603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.767574    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:56.767734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.767691    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:56.767862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.767846    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6m9p\" (UniqueName: \"kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-kube-api-access-j6m9p\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:56.767935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.767922    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:56.876416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.872896    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6m9p\" (UniqueName: \"kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-kube-api-access-j6m9p\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:56.876416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.872981    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:56.876416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.873207    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:56.876416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.873316    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:56.876416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.874319    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:56.896464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.895141    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:56.920485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.920282    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:56.961414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.961307    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6m9p\" (UniqueName: \"kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-kube-api-access-j6m9p\") pod \"payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") " pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:57.000046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:56.975262    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:57.600470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.565451    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"]
Oct 30 21:44:57.649416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.648206    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:57.683419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.660919    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"payload-processing-external-model-e2e--8e1d66cd-dockercfg-d8wks\""
Oct 30 21:44:57.683419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.661093    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"kube-root-ca.crt\""
Oct 30 21:44:57.683419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.661192    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0ea1ea\""
Oct 30 21:44:57.683419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.661870    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"routing-overlay\""
Oct 30 21:44:57.683419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.661968    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"agc-cred-00b228f0efa6\""
Oct 30 21:44:57.683419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.662032    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0ea1ea\"/\"openshift-service-ca.crt\""
Oct 30 21:44:57.783439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.775710    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:57.783439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.775766    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzz2w\" (UniqueName: \"kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-kube-api-access-nzz2w\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:57.783439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.775889    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:57.783439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.776089    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:57.911553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.877553    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:57.911553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.877790    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:57.911553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.877831    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzz2w\" (UniqueName: \"kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-kube-api-access-nzz2w\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:57.911553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.877946    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:57.911553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.888140    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:57.911553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.888740    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:57.930510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.930472    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"]
Oct 30 21:44:57.942682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.935699    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"]
Oct 30 21:44:57.966993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.966131    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:57.995205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:57.995173    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzz2w\" (UniqueName: \"kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-kube-api-access-nzz2w\") pod \"payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") " pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:58.254554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.254521    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:58.414628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.414557    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"]
Oct 30 21:44:58.420646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.420627    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.441032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.441012    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f1ed97\""
Oct 30 21:44:58.441032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.441016    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"kube-root-ca.crt\""
Oct 30 21:44:58.441217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.441202    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"routing-overlay\""
Oct 30 21:44:58.441271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.441224    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"openshift-service-ca.crt\""
Oct 30 21:44:58.446483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.446467    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"agc-cred-820a2a47e819\""
Oct 30 21:44:58.446738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.446711    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"payload-processing-external-model-e2e--650b641a-dockercfg-r6s2h\""
Oct 30 21:44:58.446810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.446767    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1ed97\"/\"agc-cred-bb672c18dcea\""
Oct 30 21:44:58.485739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.485709    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"]
Oct 30 21:44:58.504749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.504716    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.504920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.504841    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.505078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.505053    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.505146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.505098    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ncng\" (UniqueName: \"kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-kube-api-access-4ncng\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.550741 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:44:58.550714    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4fcb54_9c19_4c50_b5fb_c07455a18720.slice/crio-809a156c7643ce4473dd196cfe7818af49c0aa632df0418807849c3df9109365 WatchSource:0}: Error finding container 809a156c7643ce4473dd196cfe7818af49c0aa632df0418807849c3df9109365: Status 404 returned error can't find the container with id 809a156c7643ce4473dd196cfe7818af49c0aa632df0418807849c3df9109365
Oct 30 21:44:58.552581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.552554    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"]
Oct 30 21:44:58.607216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.607190    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.607358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.607251    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.607473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.607370    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.607473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.607409    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ncng\" (UniqueName: \"kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-kube-api-access-4ncng\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.607886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.607863    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.608086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.608069    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.609857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.609840    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.619366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.619343    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ncng\" (UniqueName: \"kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-kube-api-access-4ncng\") pod \"payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") " pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:58.668532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.668458    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9" event={"ID":"ad4fcb54-9c19-4c50-b5fb-c07455a18720","Type":"ContainerStarted","Data":"084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15"}
Oct 30 21:44:58.668532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.668517    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9" event={"ID":"ad4fcb54-9c19-4c50-b5fb-c07455a18720","Type":"ContainerStarted","Data":"809a156c7643ce4473dd196cfe7818af49c0aa632df0418807849c3df9109365"}
Oct 30 21:44:58.669016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.668593    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:44:58.670365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.670342    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs" event={"ID":"2f5bfd22-d58e-471a-aef3-565da8291106","Type":"ContainerStarted","Data":"9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172"}
Oct 30 21:44:58.670479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.670385    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs" event={"ID":"2f5bfd22-d58e-471a-aef3-565da8291106","Type":"ContainerStarted","Data":"b477349525b4adab0f010f5c2e58483f0228de6b835fdc2a443b265b6eeb2340"}
Oct 30 21:44:58.670479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.670451    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:44:58.697564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.697520    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9" podStartSLOduration=1.697507286 podStartE2EDuration="1.697507286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:58.693466737 +0000 UTC m=+2611.740025924" watchObservedRunningTime="2026-10-30 21:44:58.697507286 +0000 UTC m=+2611.744066473"
Oct 30 21:44:58.731260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.731218    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs" podStartSLOduration=2.731205782 podStartE2EDuration="2.731205782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:58.729333858 +0000 UTC m=+2611.775893021" watchObservedRunningTime="2026-10-30 21:44:58.731205782 +0000 UTC m=+2611.777764926"
Oct 30 21:44:58.735996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:58.735973    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:59.071026 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:44:59.071000    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8093b0c3_59d2_4c75_8b63_286b5ccaf1cf.slice/crio-06ee8773ee67bb9873d2c885f9cdc45b8ba25bd5357a2dff7793199d6e201483 WatchSource:0}: Error finding container 06ee8773ee67bb9873d2c885f9cdc45b8ba25bd5357a2dff7793199d6e201483: Status 404 returned error can't find the container with id 06ee8773ee67bb9873d2c885f9cdc45b8ba25bd5357a2dff7793199d6e201483
Oct 30 21:44:59.084716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:59.084692    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"]
Oct 30 21:44:59.177433 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:44:59.177356    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:44:59.681600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:59.681554    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c" event={"ID":"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf","Type":"ContainerStarted","Data":"52d6bf9175a21b874c1e002307ec178db9b4da4dec117f2fe354712f00ec9125"}
Oct 30 21:44:59.682089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:59.681618    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c" event={"ID":"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf","Type":"ContainerStarted","Data":"06ee8773ee67bb9873d2c885f9cdc45b8ba25bd5357a2dff7793199d6e201483"}
Oct 30 21:44:59.682089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:59.681978    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:44:59.709256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:44:59.709210    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c" podStartSLOduration=1.7091964480000001 podStartE2EDuration="1.709196448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:59.708324785 +0000 UTC m=+2612.754883944" watchObservedRunningTime="2026-10-30 21:44:59.709196448 +0000 UTC m=+2612.755755593"
Oct 30 21:45:00.189170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.189139    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"]
Oct 30 21:45:00.194651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.194629    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"
Oct 30 21:45:00.194789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.194629    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"]
Oct 30 21:45:00.202723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.202696    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"
Oct 30 21:45:00.232556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.232528    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:45:00.232746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.232718    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"]
Oct 30 21:45:00.239058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.239032    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"
Oct 30 21:45:00.239306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.239272    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"]
Oct 30 21:45:00.250988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.250964    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"]
Oct 30 21:45:00.254921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.254895    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"]
Oct 30 21:45:00.262637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.262569    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"]
Oct 30 21:45:00.262771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.262679    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"
Oct 30 21:45:00.271681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.271655    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"
Oct 30 21:45:00.276410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.276371    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"]
Oct 30 21:45:00.285510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.285480    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"]
Oct 30 21:45:00.290272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.290248    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"]
Oct 30 21:45:00.295060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.293274    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/409b4a57-a05d-49c5-92e0-58a4b1712afc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42\" (UID: \"409b4a57-a05d-49c5-92e0-58a4b1712afc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"
Oct 30 21:45:00.295060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.293589    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n279k\" (UniqueName: \"kubernetes.io/projected/79a3898f-6fd6-4db6-811b-48fdef8f81ba-kube-api-access-n279k\") pod \"maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z\" (UID: \"79a3898f-6fd6-4db6-811b-48fdef8f81ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"
Oct 30 21:45:00.295060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.293658    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59jv\" (UniqueName: \"kubernetes.io/projected/409b4a57-a05d-49c5-92e0-58a4b1712afc-kube-api-access-r59jv\") pod \"maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42\" (UID: \"409b4a57-a05d-49c5-92e0-58a4b1712afc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"
Oct 30 21:45:00.295060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.293746    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a3898f-6fd6-4db6-811b-48fdef8f81ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z\" (UID: \"79a3898f-6fd6-4db6-811b-48fdef8f81ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"
Oct 30 21:45:00.301564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.301539    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"]
Oct 30 21:45:00.301719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.301699    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"
Oct 30 21:45:00.303594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.303571    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"]
Oct 30 21:45:00.304806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.304785    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:45:00.304953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.304939    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:45:00.313018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.312997    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"]
Oct 30 21:45:00.313152 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.313134    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"
Oct 30 21:45:00.318468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.318447    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"]
Oct 30 21:45:00.318741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.318569    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"
Oct 30 21:45:00.332527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.332508    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"]
Oct 30 21:45:00.335505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.335484    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"
Oct 30 21:45:00.347126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.347105    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"]
Oct 30 21:45:00.347324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.347312    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"]
Oct 30 21:45:00.347493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.347214    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"
Oct 30 21:45:00.359330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.359304    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"]
Oct 30 21:45:00.366877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.366843    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"]
Oct 30 21:45:00.372697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.372665    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"]
Oct 30 21:45:00.378742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.378718    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"]
Oct 30 21:45:00.383546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.383527    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"]
Oct 30 21:45:00.383671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.383648    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"
Oct 30 21:45:00.388452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.388435    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"
Oct 30 21:45:00.389131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.389112    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"]
Oct 30 21:45:00.395797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.395717    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e008fe-d6b5-4328-a70d-188bdb928a4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr\" (UID: \"57e008fe-d6b5-4328-a70d-188bdb928a4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"
Oct 30 21:45:00.397338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.395920    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs7tb\" (UniqueName: \"kubernetes.io/projected/c3712d27-9480-45ce-a089-caff86ddb3ec-kube-api-access-fs7tb\") pod \"maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg\" (UID: \"c3712d27-9480-45ce-a089-caff86ddb3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"
Oct 30 21:45:00.397338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.395958    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c42b67a-e483-4511-9be1-005d53f02bdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd\" (UID: \"2c42b67a-e483-4511-9be1-005d53f02bdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"
Oct 30 21:45:00.397338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.395990    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3712d27-9480-45ce-a089-caff86ddb3ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg\" (UID: \"c3712d27-9480-45ce-a089-caff86ddb3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"
Oct 30 21:45:00.397338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.396033    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83727964-4998-42ed-82b2-7ac611269201-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6\" (UID: \"83727964-4998-42ed-82b2-7ac611269201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"
Oct 30 21:45:00.397338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.396075    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/409b4a57-a05d-49c5-92e0-58a4b1712afc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42\" (UID: \"409b4a57-a05d-49c5-92e0-58a4b1712afc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"
Oct 30 21:45:00.397338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.396182    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbtf4\" (UniqueName: \"kubernetes.io/projected/83727964-4998-42ed-82b2-7ac611269201-kube-api-access-pbtf4\") pod \"maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6\" (UID: \"83727964-4998-42ed-82b2-7ac611269201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"
Oct 30 21:45:00.397338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.396251    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8qcp\" (UniqueName: \"kubernetes.io/projected/57e008fe-d6b5-4328-a70d-188bdb928a4d-kube-api-access-p8qcp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr\" (UID: \"57e008fe-d6b5-4328-a70d-188bdb928a4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"
Oct 30 21:45:00.397338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.396298    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98z5g\" (UniqueName: \"kubernetes.io/projected/2c42b67a-e483-4511-9be1-005d53f02bdc-kube-api-access-98z5g\") pod \"maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd\" (UID: \"2c42b67a-e483-4511-9be1-005d53f02bdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"
Oct 30 21:45:00.397338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.396326    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n279k\" (UniqueName: \"kubernetes.io/projected/79a3898f-6fd6-4db6-811b-48fdef8f81ba-kube-api-access-n279k\") pod \"maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z\" (UID: \"79a3898f-6fd6-4db6-811b-48fdef8f81ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"
Oct 30 21:45:00.397338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.396366    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59jv\" (UniqueName: \"kubernetes.io/projected/409b4a57-a05d-49c5-92e0-58a4b1712afc-kube-api-access-r59jv\") pod \"maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42\" (UID: \"409b4a57-a05d-49c5-92e0-58a4b1712afc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"
Oct 30 21:45:00.397338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.396482    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a3898f-6fd6-4db6-811b-48fdef8f81ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z\" (UID: \"79a3898f-6fd6-4db6-811b-48fdef8f81ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"
Oct 30 21:45:00.397338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.397254    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a3898f-6fd6-4db6-811b-48fdef8f81ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z\" (UID: \"79a3898f-6fd6-4db6-811b-48fdef8f81ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"
Oct 30 21:45:00.397952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.397343    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/409b4a57-a05d-49c5-92e0-58a4b1712afc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42\" (UID: \"409b4a57-a05d-49c5-92e0-58a4b1712afc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"
Oct 30 21:45:00.399647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.399629    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"
Oct 30 21:45:00.431000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.430966    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"]
Oct 30 21:45:00.445303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.445215    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n279k\" (UniqueName: \"kubernetes.io/projected/79a3898f-6fd6-4db6-811b-48fdef8f81ba-kube-api-access-n279k\") pod \"maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z\" (UID: \"79a3898f-6fd6-4db6-811b-48fdef8f81ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"
Oct 30 21:45:00.446287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.446256    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59jv\" (UniqueName: \"kubernetes.io/projected/409b4a57-a05d-49c5-92e0-58a4b1712afc-kube-api-access-r59jv\") pod \"maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42\" (UID: \"409b4a57-a05d-49c5-92e0-58a4b1712afc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"
Oct 30 21:45:00.449679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.449652    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"
Oct 30 21:45:00.458478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.457465    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"]
Oct 30 21:45:00.460002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.459977    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"]
Oct 30 21:45:00.465620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.465595    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"]
Oct 30 21:45:00.472754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.472720    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"
Oct 30 21:45:00.483966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.483938    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"]
Oct 30 21:45:00.498112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498084    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e008fe-d6b5-4328-a70d-188bdb928a4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr\" (UID: \"57e008fe-d6b5-4328-a70d-188bdb928a4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"
Oct 30 21:45:00.498329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498123    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs7tb\" (UniqueName: \"kubernetes.io/projected/c3712d27-9480-45ce-a089-caff86ddb3ec-kube-api-access-fs7tb\") pod \"maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg\" (UID: \"c3712d27-9480-45ce-a089-caff86ddb3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"
Oct 30 21:45:00.498329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498153    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scd68\" (UniqueName: \"kubernetes.io/projected/aedc6368-2835-4821-bf6c-9cb838f21611-kube-api-access-scd68\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp\" (UID: \"aedc6368-2835-4821-bf6c-9cb838f21611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"
Oct 30 21:45:00.498329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498180    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c42b67a-e483-4511-9be1-005d53f02bdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd\" (UID: \"2c42b67a-e483-4511-9be1-005d53f02bdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"
Oct 30 21:45:00.498329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498209    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3712d27-9480-45ce-a089-caff86ddb3ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg\" (UID: \"c3712d27-9480-45ce-a089-caff86ddb3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"
Oct 30 21:45:00.498329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498235    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdqp6\" (UniqueName: \"kubernetes.io/projected/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-kube-api-access-kdqp6\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs\" (UID: \"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"
Oct 30 21:45:00.498329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498276    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83727964-4998-42ed-82b2-7ac611269201-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6\" (UID: \"83727964-4998-42ed-82b2-7ac611269201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"
Oct 30 21:45:00.498329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498308    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9d9\" (UniqueName: \"kubernetes.io/projected/a60f039f-582f-4178-916d-5bf0260ed803-kube-api-access-xf9d9\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6\" (UID: \"a60f039f-582f-4178-916d-5bf0260ed803\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"
Oct 30 21:45:00.498967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498414    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs\" (UID: \"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"
Oct 30 21:45:00.498967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498443    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cfd24a-7880-40a4-98ed-0c9326491795-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w\" (UID: \"06cfd24a-7880-40a4-98ed-0c9326491795\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"
Oct 30 21:45:00.498967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498469    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdpl9\" (UniqueName: \"kubernetes.io/projected/296ad3ea-d106-480c-b05c-2c44452d39ca-kube-api-access-vdpl9\") pod \"maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6\" (UID: \"296ad3ea-d106-480c-b05c-2c44452d39ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"
Oct 30 21:45:00.498967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498515    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/296ad3ea-d106-480c-b05c-2c44452d39ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6\" (UID: \"296ad3ea-d106-480c-b05c-2c44452d39ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"
Oct 30 21:45:00.498967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498567    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbtf4\" (UniqueName: \"kubernetes.io/projected/83727964-4998-42ed-82b2-7ac611269201-kube-api-access-pbtf4\") pod \"maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6\" (UID: \"83727964-4998-42ed-82b2-7ac611269201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"
Oct 30 21:45:00.498967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498645    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8qcp\" (UniqueName: \"kubernetes.io/projected/57e008fe-d6b5-4328-a70d-188bdb928a4d-kube-api-access-p8qcp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr\" (UID: \"57e008fe-d6b5-4328-a70d-188bdb928a4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"
Oct 30 21:45:00.498967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498697    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98z5g\" (UniqueName: \"kubernetes.io/projected/2c42b67a-e483-4511-9be1-005d53f02bdc-kube-api-access-98z5g\") pod \"maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd\" (UID: \"2c42b67a-e483-4511-9be1-005d53f02bdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"
Oct 30 21:45:00.498967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.498727    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz8wx\" (UniqueName: \"kubernetes.io/projected/5a319997-3bb4-419a-a27b-23849ee7691b-kube-api-access-hz8wx\") pod \"maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx\" (UID: \"5a319997-3bb4-419a-a27b-23849ee7691b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"
Oct 30 21:45:00.499579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.499250    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e008fe-d6b5-4328-a70d-188bdb928a4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr\" (UID: \"57e008fe-d6b5-4328-a70d-188bdb928a4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"
Oct 30 21:45:00.499810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.499785    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedc6368-2835-4821-bf6c-9cb838f21611-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp\" (UID: \"aedc6368-2835-4821-bf6c-9cb838f21611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"
Oct 30 21:45:00.499909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.499832    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a319997-3bb4-419a-a27b-23849ee7691b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx\" (UID: \"5a319997-3bb4-419a-a27b-23849ee7691b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"
Oct 30 21:45:00.500636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.500611    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c42b67a-e483-4511-9be1-005d53f02bdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd\" (UID: \"2c42b67a-e483-4511-9be1-005d53f02bdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"
Oct 30 21:45:00.500793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.500674    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3712d27-9480-45ce-a089-caff86ddb3ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg\" (UID: \"c3712d27-9480-45ce-a089-caff86ddb3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"
Oct 30 21:45:00.500793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.500674    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83727964-4998-42ed-82b2-7ac611269201-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6\" (UID: \"83727964-4998-42ed-82b2-7ac611269201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"
Oct 30 21:45:00.501739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.501437    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee00d03a-36a1-47fe-be99-0cf595abd3a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk\" (UID: \"ee00d03a-36a1-47fe-be99-0cf595abd3a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"
Oct 30 21:45:00.501739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.501591    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkxxx\" (UniqueName: \"kubernetes.io/projected/06cfd24a-7880-40a4-98ed-0c9326491795-kube-api-access-xkxxx\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w\" (UID: \"06cfd24a-7880-40a4-98ed-0c9326491795\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"
Oct 30 21:45:00.501739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.501630    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rkd4\" (UniqueName: \"kubernetes.io/projected/ee00d03a-36a1-47fe-be99-0cf595abd3a5-kube-api-access-9rkd4\") pod \"maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk\" (UID: \"ee00d03a-36a1-47fe-be99-0cf595abd3a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"
Oct 30 21:45:00.501986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.501764    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60f039f-582f-4178-916d-5bf0260ed803-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6\" (UID: \"a60f039f-582f-4178-916d-5bf0260ed803\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"
Oct 30 21:45:00.506805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.504994    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"]
Oct 30 21:45:00.515439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.512430    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"
Oct 30 21:45:00.517759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.517738    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"
Oct 30 21:45:00.526636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.526613    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"
Oct 30 21:45:00.536236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.536200    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs7tb\" (UniqueName: \"kubernetes.io/projected/c3712d27-9480-45ce-a089-caff86ddb3ec-kube-api-access-fs7tb\") pod \"maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg\" (UID: \"c3712d27-9480-45ce-a089-caff86ddb3ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"
Oct 30 21:45:00.546009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.545939    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"]
Oct 30 21:45:00.554782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.554729    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"]
Oct 30 21:45:00.559775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.556714    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbtf4\" (UniqueName: \"kubernetes.io/projected/83727964-4998-42ed-82b2-7ac611269201-kube-api-access-pbtf4\") pod \"maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6\" (UID: \"83727964-4998-42ed-82b2-7ac611269201\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"
Oct 30 21:45:00.559775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.556771    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8qcp\" (UniqueName: \"kubernetes.io/projected/57e008fe-d6b5-4328-a70d-188bdb928a4d-kube-api-access-p8qcp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr\" (UID: \"57e008fe-d6b5-4328-a70d-188bdb928a4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"
Oct 30 21:45:00.559775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.557246    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98z5g\" (UniqueName: \"kubernetes.io/projected/2c42b67a-e483-4511-9be1-005d53f02bdc-kube-api-access-98z5g\") pod \"maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd\" (UID: \"2c42b67a-e483-4511-9be1-005d53f02bdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"
Oct 30 21:45:00.566373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.566127    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"
Oct 30 21:45:00.577935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.577897    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"]
Oct 30 21:45:00.578090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.578036    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"
Oct 30 21:45:00.586063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.586034    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"]
Oct 30 21:45:00.592973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.592898    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"
Oct 30 21:45:00.603365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.603331    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee00d03a-36a1-47fe-be99-0cf595abd3a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk\" (UID: \"ee00d03a-36a1-47fe-be99-0cf595abd3a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"
Oct 30 21:45:00.603661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.603636    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9fsd\" (UniqueName: \"kubernetes.io/projected/e65d8a21-107a-40ad-b8fc-967cb5008270-kube-api-access-b9fsd\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b\" (UID: \"e65d8a21-107a-40ad-b8fc-967cb5008270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"
Oct 30 21:45:00.603810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.603791    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b82123-e7fd-41ff-9655-c2610e1973bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j\" (UID: \"f6b82123-e7fd-41ff-9655-c2610e1973bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"
Oct 30 21:45:00.604008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.603990    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkxxx\" (UniqueName: \"kubernetes.io/projected/06cfd24a-7880-40a4-98ed-0c9326491795-kube-api-access-xkxxx\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w\" (UID: \"06cfd24a-7880-40a4-98ed-0c9326491795\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"
Oct 30 21:45:00.604147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.604129    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rkd4\" (UniqueName: \"kubernetes.io/projected/ee00d03a-36a1-47fe-be99-0cf595abd3a5-kube-api-access-9rkd4\") pod \"maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk\" (UID: \"ee00d03a-36a1-47fe-be99-0cf595abd3a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"
Oct 30 21:45:00.605276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.604684    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee00d03a-36a1-47fe-be99-0cf595abd3a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk\" (UID: \"ee00d03a-36a1-47fe-be99-0cf595abd3a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"
Oct 30 21:45:00.605276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.604958    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"
Oct 30 21:45:00.605514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.605495    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65d8a21-107a-40ad-b8fc-967cb5008270-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b\" (UID: \"e65d8a21-107a-40ad-b8fc-967cb5008270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"
Oct 30 21:45:00.605695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.605677    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60f039f-582f-4178-916d-5bf0260ed803-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6\" (UID: \"a60f039f-582f-4178-916d-5bf0260ed803\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"
Oct 30 21:45:00.606301 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.606281    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qv2g\" (UniqueName: \"kubernetes.io/projected/f6b82123-e7fd-41ff-9655-c2610e1973bf-kube-api-access-9qv2g\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j\" (UID: \"f6b82123-e7fd-41ff-9655-c2610e1973bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"
Oct 30 21:45:00.606492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.606473    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8szsj\" (UniqueName: \"kubernetes.io/projected/4b80735b-3681-4774-b4a3-821f37748327-kube-api-access-8szsj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj\" (UID: \"4b80735b-3681-4774-b4a3-821f37748327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"
Oct 30 21:45:00.606712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.606693    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scd68\" (UniqueName: \"kubernetes.io/projected/aedc6368-2835-4821-bf6c-9cb838f21611-kube-api-access-scd68\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp\" (UID: \"aedc6368-2835-4821-bf6c-9cb838f21611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"
Oct 30 21:45:00.606854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.606837    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdqp6\" (UniqueName: \"kubernetes.io/projected/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-kube-api-access-kdqp6\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs\" (UID: \"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"
Oct 30 21:45:00.606975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.606959    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b80735b-3681-4774-b4a3-821f37748327-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj\" (UID: \"4b80735b-3681-4774-b4a3-821f37748327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"
Oct 30 21:45:00.607122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.607106    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9d9\" (UniqueName: \"kubernetes.io/projected/a60f039f-582f-4178-916d-5bf0260ed803-kube-api-access-xf9d9\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6\" (UID: \"a60f039f-582f-4178-916d-5bf0260ed803\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"
Oct 30 21:45:00.607293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.607278    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs\" (UID: \"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"
Oct 30 21:45:00.608653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.608631    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cfd24a-7880-40a4-98ed-0c9326491795-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w\" (UID: \"06cfd24a-7880-40a4-98ed-0c9326491795\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"
Oct 30 21:45:00.608804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.607730    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60f039f-582f-4178-916d-5bf0260ed803-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6\" (UID: \"a60f039f-582f-4178-916d-5bf0260ed803\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"
Oct 30 21:45:00.608872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.608790    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdpl9\" (UniqueName: \"kubernetes.io/projected/296ad3ea-d106-480c-b05c-2c44452d39ca-kube-api-access-vdpl9\") pod \"maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6\" (UID: \"296ad3ea-d106-480c-b05c-2c44452d39ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"
Oct 30 21:45:00.609405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.609363    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/296ad3ea-d106-480c-b05c-2c44452d39ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6\" (UID: \"296ad3ea-d106-480c-b05c-2c44452d39ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"
Oct 30 21:45:00.609665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.609646    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz8wx\" (UniqueName: \"kubernetes.io/projected/5a319997-3bb4-419a-a27b-23849ee7691b-kube-api-access-hz8wx\") pod \"maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx\" (UID: \"5a319997-3bb4-419a-a27b-23849ee7691b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"
Oct 30 21:45:00.609838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.609823    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedc6368-2835-4821-bf6c-9cb838f21611-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp\" (UID: \"aedc6368-2835-4821-bf6c-9cb838f21611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"
Oct 30 21:45:00.609949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.609936    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a319997-3bb4-419a-a27b-23849ee7691b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx\" (UID: \"5a319997-3bb4-419a-a27b-23849ee7691b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"
Oct 30 21:45:00.610225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.610203    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cfd24a-7880-40a4-98ed-0c9326491795-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w\" (UID: \"06cfd24a-7880-40a4-98ed-0c9326491795\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"
Oct 30 21:45:00.612236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.612082    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a319997-3bb4-419a-a27b-23849ee7691b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx\" (UID: \"5a319997-3bb4-419a-a27b-23849ee7691b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"
Oct 30 21:45:00.618961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.612563    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs\" (UID: \"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"
Oct 30 21:45:00.618961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.612735    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedc6368-2835-4821-bf6c-9cb838f21611-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp\" (UID: \"aedc6368-2835-4821-bf6c-9cb838f21611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"
Oct 30 21:45:00.618961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.618150    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"]
Oct 30 21:45:00.618961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.618697    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/296ad3ea-d106-480c-b05c-2c44452d39ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6\" (UID: \"296ad3ea-d106-480c-b05c-2c44452d39ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"
Oct 30 21:45:00.624430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.623520    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"
Oct 30 21:45:00.650060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.650029    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkxxx\" (UniqueName: \"kubernetes.io/projected/06cfd24a-7880-40a4-98ed-0c9326491795-kube-api-access-xkxxx\") pod \"maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w\" (UID: \"06cfd24a-7880-40a4-98ed-0c9326491795\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"
Oct 30 21:45:00.652470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.652227    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scd68\" (UniqueName: \"kubernetes.io/projected/aedc6368-2835-4821-bf6c-9cb838f21611-kube-api-access-scd68\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp\" (UID: \"aedc6368-2835-4821-bf6c-9cb838f21611\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"
Oct 30 21:45:00.653226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.653198    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz8wx\" (UniqueName: \"kubernetes.io/projected/5a319997-3bb4-419a-a27b-23849ee7691b-kube-api-access-hz8wx\") pod \"maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx\" (UID: \"5a319997-3bb4-419a-a27b-23849ee7691b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"
Oct 30 21:45:00.653326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.653281    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rkd4\" (UniqueName: \"kubernetes.io/projected/ee00d03a-36a1-47fe-be99-0cf595abd3a5-kube-api-access-9rkd4\") pod \"maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk\" (UID: \"ee00d03a-36a1-47fe-be99-0cf595abd3a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"
Oct 30 21:45:00.653565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.653512    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdpl9\" (UniqueName: \"kubernetes.io/projected/296ad3ea-d106-480c-b05c-2c44452d39ca-kube-api-access-vdpl9\") pod \"maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6\" (UID: \"296ad3ea-d106-480c-b05c-2c44452d39ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"
Oct 30 21:45:00.653886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.653860    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdqp6\" (UniqueName: \"kubernetes.io/projected/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-kube-api-access-kdqp6\") pod \"maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs\" (UID: \"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"
Oct 30 21:45:00.655859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.655832    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9d9\" (UniqueName: \"kubernetes.io/projected/a60f039f-582f-4178-916d-5bf0260ed803-kube-api-access-xf9d9\") pod \"maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6\" (UID: \"a60f039f-582f-4178-916d-5bf0260ed803\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"
Oct 30 21:45:00.672707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.672617    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"
Oct 30 21:45:00.675411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.675364    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"
Oct 30 21:45:00.681437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.681189    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"
Oct 30 21:45:00.712072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.712031    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b80735b-3681-4774-b4a3-821f37748327-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj\" (UID: \"4b80735b-3681-4774-b4a3-821f37748327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"
Oct 30 21:45:00.712467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.712264    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bba61f8-4422-4206-8723-40d9ac86a4c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7\" (UID: \"6bba61f8-4422-4206-8723-40d9ac86a4c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"
Oct 30 21:45:00.712467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.712366    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9fsd\" (UniqueName: \"kubernetes.io/projected/e65d8a21-107a-40ad-b8fc-967cb5008270-kube-api-access-b9fsd\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b\" (UID: \"e65d8a21-107a-40ad-b8fc-967cb5008270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"
Oct 30 21:45:00.712467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.712421    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b82123-e7fd-41ff-9655-c2610e1973bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j\" (UID: \"f6b82123-e7fd-41ff-9655-c2610e1973bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"
Oct 30 21:45:00.712467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.712458    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9stg\" (UniqueName: \"kubernetes.io/projected/6bba61f8-4422-4206-8723-40d9ac86a4c2-kube-api-access-r9stg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7\" (UID: \"6bba61f8-4422-4206-8723-40d9ac86a4c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"
Oct 30 21:45:00.712718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.712543    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65d8a21-107a-40ad-b8fc-967cb5008270-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b\" (UID: \"e65d8a21-107a-40ad-b8fc-967cb5008270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"
Oct 30 21:45:00.712718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.712606    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qv2g\" (UniqueName: \"kubernetes.io/projected/f6b82123-e7fd-41ff-9655-c2610e1973bf-kube-api-access-9qv2g\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j\" (UID: \"f6b82123-e7fd-41ff-9655-c2610e1973bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"
Oct 30 21:45:00.712718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.712647    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8szsj\" (UniqueName: \"kubernetes.io/projected/4b80735b-3681-4774-b4a3-821f37748327-kube-api-access-8szsj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj\" (UID: \"4b80735b-3681-4774-b4a3-821f37748327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"
Oct 30 21:45:00.713589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.712859    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b80735b-3681-4774-b4a3-821f37748327-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj\" (UID: \"4b80735b-3681-4774-b4a3-821f37748327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"
Oct 30 21:45:00.713589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.713380    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65d8a21-107a-40ad-b8fc-967cb5008270-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b\" (UID: \"e65d8a21-107a-40ad-b8fc-967cb5008270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"
Oct 30 21:45:00.713589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.713558    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b82123-e7fd-41ff-9655-c2610e1973bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j\" (UID: \"f6b82123-e7fd-41ff-9655-c2610e1973bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"
Oct 30 21:45:00.716966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.716631    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"
Oct 30 21:45:00.730128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.729900    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"
Oct 30 21:45:00.730812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.730785    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qv2g\" (UniqueName: \"kubernetes.io/projected/f6b82123-e7fd-41ff-9655-c2610e1973bf-kube-api-access-9qv2g\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j\" (UID: \"f6b82123-e7fd-41ff-9655-c2610e1973bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"
Oct 30 21:45:00.730964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.730939    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8szsj\" (UniqueName: \"kubernetes.io/projected/4b80735b-3681-4774-b4a3-821f37748327-kube-api-access-8szsj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj\" (UID: \"4b80735b-3681-4774-b4a3-821f37748327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"
Oct 30 21:45:00.738171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.738129    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9fsd\" (UniqueName: \"kubernetes.io/projected/e65d8a21-107a-40ad-b8fc-967cb5008270-kube-api-access-b9fsd\") pod \"maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b\" (UID: \"e65d8a21-107a-40ad-b8fc-967cb5008270\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"
Oct 30 21:45:00.751573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.751542    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"
Oct 30 21:45:00.824608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.824571    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bba61f8-4422-4206-8723-40d9ac86a4c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7\" (UID: \"6bba61f8-4422-4206-8723-40d9ac86a4c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"
Oct 30 21:45:00.824746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.824715    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9stg\" (UniqueName: \"kubernetes.io/projected/6bba61f8-4422-4206-8723-40d9ac86a4c2-kube-api-access-r9stg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7\" (UID: \"6bba61f8-4422-4206-8723-40d9ac86a4c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"
Oct 30 21:45:00.837236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.830066    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bba61f8-4422-4206-8723-40d9ac86a4c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7\" (UID: \"6bba61f8-4422-4206-8723-40d9ac86a4c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"
Oct 30 21:45:00.837236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.831403    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"
Oct 30 21:45:00.837236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.832410    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"
Oct 30 21:45:00.837236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.835688    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"
Oct 30 21:45:00.850010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.849939    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9stg\" (UniqueName: \"kubernetes.io/projected/6bba61f8-4422-4206-8723-40d9ac86a4c2-kube-api-access-r9stg\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7\" (UID: \"6bba61f8-4422-4206-8723-40d9ac86a4c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"
Oct 30 21:45:00.941464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.941197    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"
Oct 30 21:45:00.950935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:00.950733    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"
Oct 30 21:45:01.314238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.314201    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"]
Oct 30 21:45:01.357534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.357461    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"]
Oct 30 21:45:01.361955 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:45:01.361924    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a3898f_6fd6_4db6_811b_48fdef8f81ba.slice/crio-4393ed71decd2402d9a8de1cf29969f537798db669acae854605f85a1f88b689 WatchSource:0}: Error finding container 4393ed71decd2402d9a8de1cf29969f537798db669acae854605f85a1f88b689: Status 404 returned error can't find the container with id 4393ed71decd2402d9a8de1cf29969f537798db669acae854605f85a1f88b689
Oct 30 21:45:01.362907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.362884    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"]
Oct 30 21:45:01.363247 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:45:01.363134    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3712d27_9480_45ce_a089_caff86ddb3ec.slice/crio-408c46d7aaec7eff0aa8de6610fda04b171b9dafdfc580909a3c319bb7393599 WatchSource:0}: Error finding container 408c46d7aaec7eff0aa8de6610fda04b171b9dafdfc580909a3c319bb7393599: Status 404 returned error can't find the container with id 408c46d7aaec7eff0aa8de6610fda04b171b9dafdfc580909a3c319bb7393599
Oct 30 21:45:01.519984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.519954    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"]
Oct 30 21:45:01.524990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.524968    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"]
Oct 30 21:45:01.535613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.535591    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"]
Oct 30 21:45:01.567312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.567290    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"]
Oct 30 21:45:01.682866 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.682789    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"]
Oct 30 21:45:01.698259 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:45:01.697195    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cfd24a_7880_40a4_98ed_0c9326491795.slice/crio-a3e83cbc7495b5d9fc4a37675e71de356cf8a38b94908549118790aad765563f WatchSource:0}: Error finding container a3e83cbc7495b5d9fc4a37675e71de356cf8a38b94908549118790aad765563f: Status 404 returned error can't find the container with id a3e83cbc7495b5d9fc4a37675e71de356cf8a38b94908549118790aad765563f
Oct 30 21:45:01.730912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.730831    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6" event={"ID":"83727964-4998-42ed-82b2-7ac611269201","Type":"ContainerStarted","Data":"8fdfd944f4100ef272fa63aaa1f45fd04a4afc93d9db74f57a71ecc358828c5f"}
Oct 30 21:45:01.741711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.741595    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z" event={"ID":"79a3898f-6fd6-4db6-811b-48fdef8f81ba","Type":"ContainerStarted","Data":"4393ed71decd2402d9a8de1cf29969f537798db669acae854605f85a1f88b689"}
Oct 30 21:45:01.751668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.751588    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w" event={"ID":"06cfd24a-7880-40a4-98ed-0c9326491795","Type":"ContainerStarted","Data":"a3e83cbc7495b5d9fc4a37675e71de356cf8a38b94908549118790aad765563f"}
Oct 30 21:45:01.754660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.753545    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr" event={"ID":"57e008fe-d6b5-4328-a70d-188bdb928a4d","Type":"ContainerStarted","Data":"621e5c5a02cde2c04965cbc545759428fe27aacb26116e26479b435399003a27"}
Oct 30 21:45:01.755727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.755706    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs" event={"ID":"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af","Type":"ContainerStarted","Data":"22967884c93c500bb51b296e05b3e448ca08959dcd07852296798f6ccd5c0a8d"}
Oct 30 21:45:01.761543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.761484    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"]
Oct 30 21:45:01.767730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.766728    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"]
Oct 30 21:45:01.767730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.767182    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg" event={"ID":"c3712d27-9480-45ce-a089-caff86ddb3ec","Type":"ContainerStarted","Data":"408c46d7aaec7eff0aa8de6610fda04b171b9dafdfc580909a3c319bb7393599"}
Oct 30 21:45:01.777151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.777119    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"]
Oct 30 21:45:01.777786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.777760    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42" event={"ID":"409b4a57-a05d-49c5-92e0-58a4b1712afc","Type":"ContainerStarted","Data":"975bed7f259632bcc636c518fb24397eaaa79cc1d69a999b60cf599e91512064"}
Oct 30 21:45:01.781451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.781419    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp" event={"ID":"aedc6368-2835-4821-bf6c-9cb838f21611","Type":"ContainerStarted","Data":"0ab7724b37708f89abd37e6efc68e80ce94ba6d728319a54c2b567fa518b74ce"}
Oct 30 21:45:01.783314 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:45:01.783290    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b80735b_3681_4774_b4a3_821f37748327.slice/crio-1253e6a63f0265db42215c623307320c87fee93d72520375dcfc3e8308c742d9 WatchSource:0}: Error finding container 1253e6a63f0265db42215c623307320c87fee93d72520375dcfc3e8308c742d9: Status 404 returned error can't find the container with id 1253e6a63f0265db42215c623307320c87fee93d72520375dcfc3e8308c742d9
Oct 30 21:45:01.794959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.794935    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"]
Oct 30 21:45:01.809135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.809112    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"]
Oct 30 21:45:01.843349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.843323    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"]
Oct 30 21:45:01.890917 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:45:01.890873    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bba61f8_4422_4206_8723_40d9ac86a4c2.slice/crio-548473a7f766c119ff7087ac3be46aed7faf103373c2e5b01f349b660b321468 WatchSource:0}: Error finding container 548473a7f766c119ff7087ac3be46aed7faf103373c2e5b01f349b660b321468: Status 404 returned error can't find the container with id 548473a7f766c119ff7087ac3be46aed7faf103373c2e5b01f349b660b321468
Oct 30 21:45:01.909373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.909348    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"]
Oct 30 21:45:01.930826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.930801    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"]
Oct 30 21:45:01.936526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:01.936501    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"]
Oct 30 21:45:02.003811 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:45:02.003782    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a319997_3bb4_419a_a27b_23849ee7691b.slice/crio-c3f32d113f50428bf89cd5c63974aa483ef6d0ad55754fc774fe67f387b1efa1 WatchSource:0}: Error finding container c3f32d113f50428bf89cd5c63974aa483ef6d0ad55754fc774fe67f387b1efa1: Status 404 returned error can't find the container with id c3f32d113f50428bf89cd5c63974aa483ef6d0ad55754fc774fe67f387b1efa1
Oct 30 21:45:02.311991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.311964    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:45:02.380815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.380783    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"]
Oct 30 21:45:02.804884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.804843    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp" event={"ID":"aedc6368-2835-4821-bf6c-9cb838f21611","Type":"ContainerStarted","Data":"92ae37eff5447d8245b3c330005cc81a0f5f5e0987ed7363fb191980860341cf"}
Oct 30 21:45:02.806261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.806233    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6" event={"ID":"a60f039f-582f-4178-916d-5bf0260ed803","Type":"ContainerStarted","Data":"5db8c16347b3b4ec2bdeb561fe425ccefb1319b058d46f73943cf42fb05914c7"}
Oct 30 21:45:02.807996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.807970    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6" event={"ID":"83727964-4998-42ed-82b2-7ac611269201","Type":"ContainerStarted","Data":"9a73ffd0705af097df905e78d0a60092b29f92b960d9451ea455daa668547259"}
Oct 30 21:45:02.809999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.809830    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z" event={"ID":"79a3898f-6fd6-4db6-811b-48fdef8f81ba","Type":"ContainerStarted","Data":"63753ff2132992ec0f92c22f1aefacbf9e351aab17fe43ca974c9c76a60b4285"}
Oct 30 21:45:02.810313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.810293    4918 scope.go:122] "RemoveContainer" containerID="63753ff2132992ec0f92c22f1aefacbf9e351aab17fe43ca974c9c76a60b4285"
Oct 30 21:45:02.811575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.811550    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b" event={"ID":"e65d8a21-107a-40ad-b8fc-967cb5008270","Type":"ContainerStarted","Data":"cb34c88a58e0bb5e5dbd298efa206296dadd420201d30b47fd541d1e97341ef8"}
Oct 30 21:45:02.813174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.813152    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx" event={"ID":"5a319997-3bb4-419a-a27b-23849ee7691b","Type":"ContainerStarted","Data":"c3f32d113f50428bf89cd5c63974aa483ef6d0ad55754fc774fe67f387b1efa1"}
Oct 30 21:45:02.816032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.816007    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j" event={"ID":"f6b82123-e7fd-41ff-9655-c2610e1973bf","Type":"ContainerStarted","Data":"4c6619e1240af99aa4490920a800f0bc4ccf13f0f4cfe2684ed51dc5c7560955"}
Oct 30 21:45:02.817886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.817861    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr" event={"ID":"57e008fe-d6b5-4328-a70d-188bdb928a4d","Type":"ContainerStarted","Data":"4a9b3203b0e3b030779a006eaeeaba9f04ebd84f8f30d308050b0b0db21e0a19"}
Oct 30 21:45:02.821777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.821757    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs" event={"ID":"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af","Type":"ContainerStarted","Data":"b0bdcf9a7e00b0ca3319a9ebe87c7f5a5c49844197d2588eb701afde8589d735"}
Oct 30 21:45:02.824318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.824295    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd" event={"ID":"2c42b67a-e483-4511-9be1-005d53f02bdc","Type":"ContainerStarted","Data":"42c70031e93d8e5946ca4d99d4b2ecbc0fb19fddba99125cd3ad7c80e9da2426"}
Oct 30 21:45:02.824445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.824350    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd" event={"ID":"2c42b67a-e483-4511-9be1-005d53f02bdc","Type":"ContainerStarted","Data":"210a52bd90854dfd23dd162912cb7be29b1db48381ee8d451eab2abd5dd688b6"}
Oct 30 21:45:02.826270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.826190    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj" event={"ID":"4b80735b-3681-4774-b4a3-821f37748327","Type":"ContainerStarted","Data":"ba61cd5a2340ec201bec7c05bd1431a13a316d0245714bab4683cb4050c10943"}
Oct 30 21:45:02.826270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.826235    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj" event={"ID":"4b80735b-3681-4774-b4a3-821f37748327","Type":"ContainerStarted","Data":"1253e6a63f0265db42215c623307320c87fee93d72520375dcfc3e8308c742d9"}
Oct 30 21:45:02.828739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.828689    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp" podStartSLOduration=2.828673355 podStartE2EDuration="2.828673355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.827636557 +0000 UTC m=+2615.874195725" watchObservedRunningTime="2026-10-30 21:45:02.828673355 +0000 UTC m=+2615.875232500"
Oct 30 21:45:02.832819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.832782    4918 generic.go:356] "Generic (PLEG): container finished" podID="c3712d27-9480-45ce-a089-caff86ddb3ec" containerID="1642dc98319c29ce3e6dab8e3dec6e3c16c30dde078b353fa66ac8d84d2462fa" exitCode=60
Oct 30 21:45:02.832919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.832831    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg" event={"ID":"c3712d27-9480-45ce-a089-caff86ddb3ec","Type":"ContainerDied","Data":"1642dc98319c29ce3e6dab8e3dec6e3c16c30dde078b353fa66ac8d84d2462fa"}
Oct 30 21:45:02.833089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.833072    4918 scope.go:122] "RemoveContainer" containerID="1642dc98319c29ce3e6dab8e3dec6e3c16c30dde078b353fa66ac8d84d2462fa"
Oct 30 21:45:02.835159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.835137    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6" event={"ID":"296ad3ea-d106-480c-b05c-2c44452d39ca","Type":"ContainerStarted","Data":"4499772b60e0366408a6474fd3a09937f7e4c93f72fd5263047fd15801975027"}
Oct 30 21:45:02.837281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.837257    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42" event={"ID":"409b4a57-a05d-49c5-92e0-58a4b1712afc","Type":"ContainerStarted","Data":"8152e08d518a530ca604ef428873d19a054f8ab14c3473425dd4e88d0b065a43"}
Oct 30 21:45:02.839102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.839024    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk" event={"ID":"ee00d03a-36a1-47fe-be99-0cf595abd3a5","Type":"ContainerStarted","Data":"ae56af92599b4daad04d32f1ba590f2b66a5716f271566c6361b8c6e391ae0d8"}
Oct 30 21:45:02.839102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.839070    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk" event={"ID":"ee00d03a-36a1-47fe-be99-0cf595abd3a5","Type":"ContainerStarted","Data":"b9868bc52c48d75df907ff9ee155caa73c24f562593b79444b4cd05decfdfbc2"}
Oct 30 21:45:02.841280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.841260    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7" event={"ID":"6bba61f8-4422-4206-8723-40d9ac86a4c2","Type":"ContainerStarted","Data":"f26ac3fe35f146da760e0900de8035eac581d660b803e9bdb39beaab089c9996"}
Oct 30 21:45:02.841374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.841292    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7" event={"ID":"6bba61f8-4422-4206-8723-40d9ac86a4c2","Type":"ContainerStarted","Data":"548473a7f766c119ff7087ac3be46aed7faf103373c2e5b01f349b660b321468"}
Oct 30 21:45:02.853235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.853191    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr" podStartSLOduration=2.853176517 podStartE2EDuration="2.853176517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.850184486 +0000 UTC m=+2615.896743644" watchObservedRunningTime="2026-10-30 21:45:02.853176517 +0000 UTC m=+2615.899735663"
Oct 30 21:45:02.902469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.902411    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj" podStartSLOduration=2.902374396 podStartE2EDuration="2.902374396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.884915731 +0000 UTC m=+2615.931474877" watchObservedRunningTime="2026-10-30 21:45:02.902374396 +0000 UTC m=+2615.948933541"
Oct 30 21:45:02.912438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.912337    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd" podStartSLOduration=2.912323099 podStartE2EDuration="2.912323099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.911757659 +0000 UTC m=+2615.958316824" watchObservedRunningTime="2026-10-30 21:45:02.912323099 +0000 UTC m=+2615.958882243"
Oct 30 21:45:02.955090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:02.955038    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs" podStartSLOduration=2.955024447 podStartE2EDuration="2.955024447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.949612453 +0000 UTC m=+2615.996171620" watchObservedRunningTime="2026-10-30 21:45:02.955024447 +0000 UTC m=+2616.001583621"
Oct 30 21:45:03.190751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:03.143279    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6" podStartSLOduration=3.14325601 podStartE2EDuration="3.14325601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.998900399 +0000 UTC m=+2616.045459565" watchObservedRunningTime="2026-10-30 21:45:03.14325601 +0000 UTC m=+2616.189815155"
Oct 30 21:45:03.190751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:03.155648    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk" podStartSLOduration=3.155630524 podStartE2EDuration="3.155630524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:03.089843023 +0000 UTC m=+2616.136402182" watchObservedRunningTime="2026-10-30 21:45:03.155630524 +0000 UTC m=+2616.202189671"
Oct 30 21:45:03.255761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:03.249852    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7" podStartSLOduration=3.249832985 podStartE2EDuration="3.249832985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:03.12991001 +0000 UTC m=+2616.176469157" watchObservedRunningTime="2026-10-30 21:45:03.249832985 +0000 UTC m=+2616.296392162"
Oct 30 21:45:03.255761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:03.252270    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42" podStartSLOduration=3.252257134 podStartE2EDuration="3.252257134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:03.169002856 +0000 UTC m=+2616.215561999" watchObservedRunningTime="2026-10-30 21:45:03.252257134 +0000 UTC m=+2616.298816279"
Oct 30 21:45:04.270844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.269792    4918 generic.go:356] "Generic (PLEG): container finished" podID="aedc6368-2835-4821-bf6c-9cb838f21611" containerID="92ae37eff5447d8245b3c330005cc81a0f5f5e0987ed7363fb191980860341cf" exitCode=60
Oct 30 21:45:04.270844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.269866    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp" event={"ID":"aedc6368-2835-4821-bf6c-9cb838f21611","Type":"ContainerDied","Data":"92ae37eff5447d8245b3c330005cc81a0f5f5e0987ed7363fb191980860341cf"}
Oct 30 21:45:04.270844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.270288    4918 scope.go:122] "RemoveContainer" containerID="92ae37eff5447d8245b3c330005cc81a0f5f5e0987ed7363fb191980860341cf"
Oct 30 21:45:04.278885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.278862    4918 generic.go:356] "Generic (PLEG): container finished" podID="83727964-4998-42ed-82b2-7ac611269201" containerID="9a73ffd0705af097df905e78d0a60092b29f92b960d9451ea455daa668547259" exitCode=60
Oct 30 21:45:04.279015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.278988    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6" event={"ID":"83727964-4998-42ed-82b2-7ac611269201","Type":"ContainerDied","Data":"9a73ffd0705af097df905e78d0a60092b29f92b960d9451ea455daa668547259"}
Oct 30 21:45:04.280033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.279324    4918 scope.go:122] "RemoveContainer" containerID="9a73ffd0705af097df905e78d0a60092b29f92b960d9451ea455daa668547259"
Oct 30 21:45:04.281297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.281276    4918 generic.go:356] "Generic (PLEG): container finished" podID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" containerID="63753ff2132992ec0f92c22f1aefacbf9e351aab17fe43ca974c9c76a60b4285" exitCode=60
Oct 30 21:45:04.281416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.281340    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z" event={"ID":"79a3898f-6fd6-4db6-811b-48fdef8f81ba","Type":"ContainerDied","Data":"63753ff2132992ec0f92c22f1aefacbf9e351aab17fe43ca974c9c76a60b4285"}
Oct 30 21:45:04.281416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.281379    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z" event={"ID":"79a3898f-6fd6-4db6-811b-48fdef8f81ba","Type":"ContainerStarted","Data":"7fd3c1973498c040ec81d0e2f3f8230fa043192ce6f3e458b6d0321e379b6e40"}
Oct 30 21:45:04.281532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.281412    4918 scope.go:122] "RemoveContainer" containerID="63753ff2132992ec0f92c22f1aefacbf9e351aab17fe43ca974c9c76a60b4285"
Oct 30 21:45:04.281663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.281646    4918 scope.go:122] "RemoveContainer" containerID="7fd3c1973498c040ec81d0e2f3f8230fa043192ce6f3e458b6d0321e379b6e40"
Oct 30 21:45:04.281956 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:04.281938    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z_odh-ai-gateway-infra(79a3898f-6fd6-4db6-811b-48fdef8f81ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba"
Oct 30 21:45:04.283323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.283302    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b" event={"ID":"e65d8a21-107a-40ad-b8fc-967cb5008270","Type":"ContainerStarted","Data":"03c8cc09628a09ca107f16f27688b60db7d321c5742cfef265d8e3771a6778f0"}
Oct 30 21:45:04.283591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.283576    4918 scope.go:122] "RemoveContainer" containerID="03c8cc09628a09ca107f16f27688b60db7d321c5742cfef265d8e3771a6778f0"
Oct 30 21:45:04.291793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.285338    4918 generic.go:356] "Generic (PLEG): container finished" podID="5a319997-3bb4-419a-a27b-23849ee7691b" containerID="bae53ac8738dd2bf55cd472f9c6d19fb005f3cc452c639402f27ecfd797d5037" exitCode=60
Oct 30 21:45:04.291793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.285469    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx" event={"ID":"5a319997-3bb4-419a-a27b-23849ee7691b","Type":"ContainerDied","Data":"bae53ac8738dd2bf55cd472f9c6d19fb005f3cc452c639402f27ecfd797d5037"}
Oct 30 21:45:04.291793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.285779    4918 scope.go:122] "RemoveContainer" containerID="bae53ac8738dd2bf55cd472f9c6d19fb005f3cc452c639402f27ecfd797d5037"
Oct 30 21:45:04.319539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.302568    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w" event={"ID":"06cfd24a-7880-40a4-98ed-0c9326491795","Type":"ContainerStarted","Data":"062f439824833622dd591bdfcd28fcc2986def41ce5ffb55df50075f39e81981"}
Oct 30 21:45:04.382413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.379367    4918 generic.go:356] "Generic (PLEG): container finished" podID="57e008fe-d6b5-4328-a70d-188bdb928a4d" containerID="4a9b3203b0e3b030779a006eaeeaba9f04ebd84f8f30d308050b0b0db21e0a19" exitCode=60
Oct 30 21:45:04.382413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.379603    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr" event={"ID":"57e008fe-d6b5-4328-a70d-188bdb928a4d","Type":"ContainerDied","Data":"4a9b3203b0e3b030779a006eaeeaba9f04ebd84f8f30d308050b0b0db21e0a19"}
Oct 30 21:45:04.382413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.381225    4918 scope.go:122] "RemoveContainer" containerID="4a9b3203b0e3b030779a006eaeeaba9f04ebd84f8f30d308050b0b0db21e0a19"
Oct 30 21:45:04.391425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.383120    4918 generic.go:356] "Generic (PLEG): container finished" podID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" containerID="b0bdcf9a7e00b0ca3319a9ebe87c7f5a5c49844197d2588eb701afde8589d735" exitCode=60
Oct 30 21:45:04.391425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.383437    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs" event={"ID":"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af","Type":"ContainerDied","Data":"b0bdcf9a7e00b0ca3319a9ebe87c7f5a5c49844197d2588eb701afde8589d735"}
Oct 30 21:45:04.391425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.385726    4918 scope.go:122] "RemoveContainer" containerID="b0bdcf9a7e00b0ca3319a9ebe87c7f5a5c49844197d2588eb701afde8589d735"
Oct 30 21:45:04.391425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.387243    4918 generic.go:356] "Generic (PLEG): container finished" podID="4b80735b-3681-4774-b4a3-821f37748327" containerID="ba61cd5a2340ec201bec7c05bd1431a13a316d0245714bab4683cb4050c10943" exitCode=60
Oct 30 21:45:04.391425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.387344    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj" event={"ID":"4b80735b-3681-4774-b4a3-821f37748327","Type":"ContainerDied","Data":"ba61cd5a2340ec201bec7c05bd1431a13a316d0245714bab4683cb4050c10943"}
Oct 30 21:45:04.419679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.419196    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg" event={"ID":"c3712d27-9480-45ce-a089-caff86ddb3ec","Type":"ContainerStarted","Data":"e65069a6f0c0a70ff185401140b9f1ea9c6cc6d677d45041f5ab4c27671e0843"}
Oct 30 21:45:04.429165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.429116    4918 generic.go:356] "Generic (PLEG): container finished" podID="409b4a57-a05d-49c5-92e0-58a4b1712afc" containerID="8152e08d518a530ca604ef428873d19a054f8ab14c3473425dd4e88d0b065a43" exitCode=60
Oct 30 21:45:04.429363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.429346    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42" event={"ID":"409b4a57-a05d-49c5-92e0-58a4b1712afc","Type":"ContainerDied","Data":"8152e08d518a530ca604ef428873d19a054f8ab14c3473425dd4e88d0b065a43"}
Oct 30 21:45:04.434605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.434576    4918 generic.go:356] "Generic (PLEG): container finished" podID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" containerID="ae56af92599b4daad04d32f1ba590f2b66a5716f271566c6361b8c6e391ae0d8" exitCode=60
Oct 30 21:45:04.434724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.434692    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk" event={"ID":"ee00d03a-36a1-47fe-be99-0cf595abd3a5","Type":"ContainerDied","Data":"ae56af92599b4daad04d32f1ba590f2b66a5716f271566c6361b8c6e391ae0d8"}
Oct 30 21:45:04.436309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.436287    4918 scope.go:122] "RemoveContainer" containerID="ba61cd5a2340ec201bec7c05bd1431a13a316d0245714bab4683cb4050c10943"
Oct 30 21:45:04.438293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.438269    4918 scope.go:122] "RemoveContainer" containerID="8152e08d518a530ca604ef428873d19a054f8ab14c3473425dd4e88d0b065a43"
Oct 30 21:45:04.438409 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.438353    4918 scope.go:122] "RemoveContainer" containerID="ae56af92599b4daad04d32f1ba590f2b66a5716f271566c6361b8c6e391ae0d8"
Oct 30 21:45:04.445261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.445226    4918 generic.go:356] "Generic (PLEG): container finished" podID="6bba61f8-4422-4206-8723-40d9ac86a4c2" containerID="f26ac3fe35f146da760e0900de8035eac581d660b803e9bdb39beaab089c9996" exitCode=60
Oct 30 21:45:04.445549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.445446    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7" event={"ID":"6bba61f8-4422-4206-8723-40d9ac86a4c2","Type":"ContainerDied","Data":"f26ac3fe35f146da760e0900de8035eac581d660b803e9bdb39beaab089c9996"}
Oct 30 21:45:04.447204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.447187    4918 scope.go:122] "RemoveContainer" containerID="f26ac3fe35f146da760e0900de8035eac581d660b803e9bdb39beaab089c9996"
Oct 30 21:45:04.463435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.463367    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j" podStartSLOduration=4.463355191 podStartE2EDuration="4.463355191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.459206311 +0000 UTC m=+2617.505765475" watchObservedRunningTime="2026-10-30 21:45:04.463355191 +0000 UTC m=+2617.509914364"
Oct 30 21:45:04.524112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.524084    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"]
Oct 30 21:45:04.528091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.528058    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:45:04.566950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.566869    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6" podStartSLOduration=4.566857149 podStartE2EDuration="4.566857149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.564793219 +0000 UTC m=+2617.611352366" watchObservedRunningTime="2026-10-30 21:45:04.566857149 +0000 UTC m=+2617.613416294"
Oct 30 21:45:04.567304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.567283    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w" podStartSLOduration=4.567276322 podStartE2EDuration="4.567276322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.535250966 +0000 UTC m=+2617.581810092" watchObservedRunningTime="2026-10-30 21:45:04.567276322 +0000 UTC m=+2617.613835466"
Oct 30 21:45:04.600667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:04.596286    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg" podStartSLOduration=4.596267343 podStartE2EDuration="4.596267343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.590299005 +0000 UTC m=+2617.636858150" watchObservedRunningTime="2026-10-30 21:45:04.596267343 +0000 UTC m=+2617.642826489"
Oct 30 21:45:05.559418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.528674    4918 generic.go:356] "Generic (PLEG): container finished" podID="2c42b67a-e483-4511-9be1-005d53f02bdc" containerID="42c70031e93d8e5946ca4d99d4b2ecbc0fb19fddba99125cd3ad7c80e9da2426" exitCode=60
Oct 30 21:45:05.559418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.528780    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd" event={"ID":"2c42b67a-e483-4511-9be1-005d53f02bdc","Type":"ContainerDied","Data":"42c70031e93d8e5946ca4d99d4b2ecbc0fb19fddba99125cd3ad7c80e9da2426"}
Oct 30 21:45:05.559418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.529332    4918 scope.go:122] "RemoveContainer" containerID="42c70031e93d8e5946ca4d99d4b2ecbc0fb19fddba99125cd3ad7c80e9da2426"
Oct 30 21:45:05.609995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.606292    4918 generic.go:356] "Generic (PLEG): container finished" podID="c3712d27-9480-45ce-a089-caff86ddb3ec" containerID="e65069a6f0c0a70ff185401140b9f1ea9c6cc6d677d45041f5ab4c27671e0843" exitCode=60
Oct 30 21:45:05.609995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.606491    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg" event={"ID":"c3712d27-9480-45ce-a089-caff86ddb3ec","Type":"ContainerDied","Data":"e65069a6f0c0a70ff185401140b9f1ea9c6cc6d677d45041f5ab4c27671e0843"}
Oct 30 21:45:05.609995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.606557    4918 scope.go:122] "RemoveContainer" containerID="1642dc98319c29ce3e6dab8e3dec6e3c16c30dde078b353fa66ac8d84d2462fa"
Oct 30 21:45:05.617439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.616348    4918 scope.go:122] "RemoveContainer" containerID="e65069a6f0c0a70ff185401140b9f1ea9c6cc6d677d45041f5ab4c27671e0843"
Oct 30 21:45:05.635690 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:05.617376    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg_odh-ai-gateway-infra(c3712d27-9480-45ce-a089-caff86ddb3ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg" podUID="c3712d27-9480-45ce-a089-caff86ddb3ec"
Oct 30 21:45:05.635690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.635205    4918 generic.go:356] "Generic (PLEG): container finished" podID="296ad3ea-d106-480c-b05c-2c44452d39ca" containerID="fe9180e53514257dbbfd9536c59b01f848b8769ef8fde68ca74a40de717eeb33" exitCode=60
Oct 30 21:45:05.635690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.635443    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6" event={"ID":"296ad3ea-d106-480c-b05c-2c44452d39ca","Type":"ContainerDied","Data":"fe9180e53514257dbbfd9536c59b01f848b8769ef8fde68ca74a40de717eeb33"}
Oct 30 21:45:05.642412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.638892    4918 scope.go:122] "RemoveContainer" containerID="fe9180e53514257dbbfd9536c59b01f848b8769ef8fde68ca74a40de717eeb33"
Oct 30 21:45:05.709096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.705692    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp" event={"ID":"aedc6368-2835-4821-bf6c-9cb838f21611","Type":"ContainerStarted","Data":"1300cc02b07c713f550535f2dde2ca35a083e9fb7302e7ba0e5420ff6000d611"}
Oct 30 21:45:05.715350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.713712    4918 generic.go:356] "Generic (PLEG): container finished" podID="a60f039f-582f-4178-916d-5bf0260ed803" containerID="76a5131da70d8e9d758c731d7414cda2895fd02b0d288323914181482da8ee83" exitCode=60
Oct 30 21:45:05.715350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.713915    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6" event={"ID":"a60f039f-582f-4178-916d-5bf0260ed803","Type":"ContainerDied","Data":"76a5131da70d8e9d758c731d7414cda2895fd02b0d288323914181482da8ee83"}
Oct 30 21:45:05.715350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.714445    4918 scope.go:122] "RemoveContainer" containerID="76a5131da70d8e9d758c731d7414cda2895fd02b0d288323914181482da8ee83"
Oct 30 21:45:05.722013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.721984    4918 generic.go:356] "Generic (PLEG): container finished" podID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" containerID="7fd3c1973498c040ec81d0e2f3f8230fa043192ce6f3e458b6d0321e379b6e40" exitCode=60
Oct 30 21:45:05.722136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.722067    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z" event={"ID":"79a3898f-6fd6-4db6-811b-48fdef8f81ba","Type":"ContainerDied","Data":"7fd3c1973498c040ec81d0e2f3f8230fa043192ce6f3e458b6d0321e379b6e40"}
Oct 30 21:45:05.722717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.722695    4918 scope.go:122] "RemoveContainer" containerID="7fd3c1973498c040ec81d0e2f3f8230fa043192ce6f3e458b6d0321e379b6e40"
Oct 30 21:45:05.723486 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:05.723158    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z_odh-ai-gateway-infra(79a3898f-6fd6-4db6-811b-48fdef8f81ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba"
Oct 30 21:45:05.726651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.726465    4918 generic.go:356] "Generic (PLEG): container finished" podID="e65d8a21-107a-40ad-b8fc-967cb5008270" containerID="03c8cc09628a09ca107f16f27688b60db7d321c5742cfef265d8e3771a6778f0" exitCode=60
Oct 30 21:45:05.726651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.726553    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b" event={"ID":"e65d8a21-107a-40ad-b8fc-967cb5008270","Type":"ContainerDied","Data":"03c8cc09628a09ca107f16f27688b60db7d321c5742cfef265d8e3771a6778f0"}
Oct 30 21:45:05.737467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.735314    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx" event={"ID":"5a319997-3bb4-419a-a27b-23849ee7691b","Type":"ContainerStarted","Data":"6dd25e3e03dce0854383c9fe95c1d35112a07e2541811710850b4c6bdfd84232"}
Oct 30 21:45:05.741767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.741745    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"]
Oct 30 21:45:05.745047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.745029    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:45:05.745263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.745249    4918 generic.go:356] "Generic (PLEG): container finished" podID="f6b82123-e7fd-41ff-9655-c2610e1973bf" containerID="e1444b740da819b032cf43dee3f8e8de8110a35e6e0c902ce85237960492c2ec" exitCode=60
Oct 30 21:45:05.745432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.745418    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j" event={"ID":"f6b82123-e7fd-41ff-9655-c2610e1973bf","Type":"ContainerDied","Data":"e1444b740da819b032cf43dee3f8e8de8110a35e6e0c902ce85237960492c2ec"}
Oct 30 21:45:05.745780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.745767    4918 scope.go:122] "RemoveContainer" containerID="e1444b740da819b032cf43dee3f8e8de8110a35e6e0c902ce85237960492c2ec"
Oct 30 21:45:05.753343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.753317    4918 generic.go:356] "Generic (PLEG): container finished" podID="06cfd24a-7880-40a4-98ed-0c9326491795" containerID="062f439824833622dd591bdfcd28fcc2986def41ce5ffb55df50075f39e81981" exitCode=60
Oct 30 21:45:05.753446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.753428    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w" event={"ID":"06cfd24a-7880-40a4-98ed-0c9326491795","Type":"ContainerDied","Data":"062f439824833622dd591bdfcd28fcc2986def41ce5ffb55df50075f39e81981"}
Oct 30 21:45:05.753803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.753783    4918 scope.go:122] "RemoveContainer" containerID="062f439824833622dd591bdfcd28fcc2986def41ce5ffb55df50075f39e81981"
Oct 30 21:45:05.970711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:05.970662    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx" podStartSLOduration=5.970646533 podStartE2EDuration="5.970646533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:05.964404926 +0000 UTC m=+2619.010964068" watchObservedRunningTime="2026-10-30 21:45:05.970646533 +0000 UTC m=+2619.017205678"
Oct 30 21:45:06.160579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.159041    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b" podStartSLOduration=6.15901935 podStartE2EDuration="6.15901935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:06.101696155 +0000 UTC m=+2619.148255301" watchObservedRunningTime="2026-10-30 21:45:06.15901935 +0000 UTC m=+2619.205578537"
Oct 30 21:45:06.772382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.772354    4918 generic.go:356] "Generic (PLEG): container finished" podID="aedc6368-2835-4821-bf6c-9cb838f21611" containerID="1300cc02b07c713f550535f2dde2ca35a083e9fb7302e7ba0e5420ff6000d611" exitCode=60
Oct 30 21:45:06.772813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.772454    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp" event={"ID":"aedc6368-2835-4821-bf6c-9cb838f21611","Type":"ContainerDied","Data":"1300cc02b07c713f550535f2dde2ca35a083e9fb7302e7ba0e5420ff6000d611"}
Oct 30 21:45:06.772813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.772507    4918 scope.go:122] "RemoveContainer" containerID="92ae37eff5447d8245b3c330005cc81a0f5f5e0987ed7363fb191980860341cf"
Oct 30 21:45:06.772971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.772825    4918 scope.go:122] "RemoveContainer" containerID="1300cc02b07c713f550535f2dde2ca35a083e9fb7302e7ba0e5420ff6000d611"
Oct 30 21:45:06.773305 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:06.773278    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp_odh-ai-gateway-infra(aedc6368-2835-4821-bf6c-9cb838f21611)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp" podUID="aedc6368-2835-4821-bf6c-9cb838f21611"
Oct 30 21:45:06.780922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.780890    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6" event={"ID":"a60f039f-582f-4178-916d-5bf0260ed803","Type":"ContainerStarted","Data":"d7943d39b1c8c61643c668cb64ceaebbfa2894011129d254b6daa4a21a2353bd"}
Oct 30 21:45:06.784784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.784760    4918 generic.go:356] "Generic (PLEG): container finished" podID="83727964-4998-42ed-82b2-7ac611269201" containerID="8db3775c6566b63512f2f53c9e28e49f253da12ebd228d469960fd932ddf95c4" exitCode=60
Oct 30 21:45:06.784942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.784800    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6" event={"ID":"83727964-4998-42ed-82b2-7ac611269201","Type":"ContainerDied","Data":"8db3775c6566b63512f2f53c9e28e49f253da12ebd228d469960fd932ddf95c4"}
Oct 30 21:45:06.785148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.785130    4918 scope.go:122] "RemoveContainer" containerID="8db3775c6566b63512f2f53c9e28e49f253da12ebd228d469960fd932ddf95c4"
Oct 30 21:45:06.785490 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:06.785450    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6_odh-ai-gateway-infra(83727964-4998-42ed-82b2-7ac611269201)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6" podUID="83727964-4998-42ed-82b2-7ac611269201"
Oct 30 21:45:06.791711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.791689    4918 generic.go:356] "Generic (PLEG): container finished" podID="e65d8a21-107a-40ad-b8fc-967cb5008270" containerID="5a0558999cf6e2c2eead69207f38e48a9dad7d1abc3fec82969047ef3c3c1d61" exitCode=60
Oct 30 21:45:06.791820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.791769    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b" event={"ID":"e65d8a21-107a-40ad-b8fc-967cb5008270","Type":"ContainerDied","Data":"5a0558999cf6e2c2eead69207f38e48a9dad7d1abc3fec82969047ef3c3c1d61"}
Oct 30 21:45:06.792019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.792003    4918 scope.go:122] "RemoveContainer" containerID="5a0558999cf6e2c2eead69207f38e48a9dad7d1abc3fec82969047ef3c3c1d61"
Oct 30 21:45:06.792304 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:06.792283    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b_odh-ai-gateway-infra(e65d8a21-107a-40ad-b8fc-967cb5008270)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b" podUID="e65d8a21-107a-40ad-b8fc-967cb5008270"
Oct 30 21:45:06.792794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.792780    4918 scope.go:122] "RemoveContainer" containerID="9a73ffd0705af097df905e78d0a60092b29f92b960d9451ea455daa668547259"
Oct 30 21:45:06.793768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.793748    4918 generic.go:356] "Generic (PLEG): container finished" podID="5a319997-3bb4-419a-a27b-23849ee7691b" containerID="6dd25e3e03dce0854383c9fe95c1d35112a07e2541811710850b4c6bdfd84232" exitCode=60
Oct 30 21:45:06.793847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.793830    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx" event={"ID":"5a319997-3bb4-419a-a27b-23849ee7691b","Type":"ContainerDied","Data":"6dd25e3e03dce0854383c9fe95c1d35112a07e2541811710850b4c6bdfd84232"}
Oct 30 21:45:06.794112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.794095    4918 scope.go:122] "RemoveContainer" containerID="6dd25e3e03dce0854383c9fe95c1d35112a07e2541811710850b4c6bdfd84232"
Oct 30 21:45:06.794371 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:06.794352    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx_odh-ai-gateway-infra(5a319997-3bb4-419a-a27b-23849ee7691b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx" podUID="5a319997-3bb4-419a-a27b-23849ee7691b"
Oct 30 21:45:06.795969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.795948    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j" event={"ID":"f6b82123-e7fd-41ff-9655-c2610e1973bf","Type":"ContainerStarted","Data":"15a956bc024235f56ecbfdef23b6848a75252a7c1dc9738c290e14cbd94ed0a9"}
Oct 30 21:45:06.797750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.797730    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w" event={"ID":"06cfd24a-7880-40a4-98ed-0c9326491795","Type":"ContainerStarted","Data":"0679ac3a391382a406b51965d332fc1d35113e00567b36fd783b42a279a378a1"}
Oct 30 21:45:06.799673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.799654    4918 generic.go:356] "Generic (PLEG): container finished" podID="57e008fe-d6b5-4328-a70d-188bdb928a4d" containerID="e4e09214c9a7354ae230fee9ce6d438787a3d451188d1a388e24135ac8685940" exitCode=60
Oct 30 21:45:06.799778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.799720    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr" event={"ID":"57e008fe-d6b5-4328-a70d-188bdb928a4d","Type":"ContainerDied","Data":"e4e09214c9a7354ae230fee9ce6d438787a3d451188d1a388e24135ac8685940"}
Oct 30 21:45:06.800013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.799997    4918 scope.go:122] "RemoveContainer" containerID="e4e09214c9a7354ae230fee9ce6d438787a3d451188d1a388e24135ac8685940"
Oct 30 21:45:06.800277 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:06.800253    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr_odh-ai-gateway-infra(57e008fe-d6b5-4328-a70d-188bdb928a4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr" podUID="57e008fe-d6b5-4328-a70d-188bdb928a4d"
Oct 30 21:45:06.801565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.801545    4918 generic.go:356] "Generic (PLEG): container finished" podID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" containerID="8cfa7a50db8b7e393a04285f412301ccb3892dbc1beff4050d49c0481c0b062c" exitCode=60
Oct 30 21:45:06.801653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.801626    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs" event={"ID":"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af","Type":"ContainerDied","Data":"8cfa7a50db8b7e393a04285f412301ccb3892dbc1beff4050d49c0481c0b062c"}
Oct 30 21:45:06.801856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.801839    4918 scope.go:122] "RemoveContainer" containerID="8cfa7a50db8b7e393a04285f412301ccb3892dbc1beff4050d49c0481c0b062c"
Oct 30 21:45:06.802169 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:06.802146    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs_odh-ai-gateway-infra(e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af"
Oct 30 21:45:06.803683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.803660    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd" event={"ID":"2c42b67a-e483-4511-9be1-005d53f02bdc","Type":"ContainerStarted","Data":"66db99a02ad0afbfc774e06d30c398595426ab1cf161c7c2bc7f64b0e0a37fa0"}
Oct 30 21:45:06.805660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.805641    4918 generic.go:356] "Generic (PLEG): container finished" podID="4b80735b-3681-4774-b4a3-821f37748327" containerID="b83481189ef09bda65f1973b330e1405df7fe27e45d4fdbd82279bd35904d936" exitCode=60
Oct 30 21:45:06.805752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.805716    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj" event={"ID":"4b80735b-3681-4774-b4a3-821f37748327","Type":"ContainerDied","Data":"b83481189ef09bda65f1973b330e1405df7fe27e45d4fdbd82279bd35904d936"}
Oct 30 21:45:06.806005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.805988    4918 scope.go:122] "RemoveContainer" containerID="b83481189ef09bda65f1973b330e1405df7fe27e45d4fdbd82279bd35904d936"
Oct 30 21:45:06.806299 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:06.806278    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj_odh-ai-gateway-infra(4b80735b-3681-4774-b4a3-821f37748327)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj" podUID="4b80735b-3681-4774-b4a3-821f37748327"
Oct 30 21:45:06.807689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.807673    4918 scope.go:122] "RemoveContainer" containerID="e65069a6f0c0a70ff185401140b9f1ea9c6cc6d677d45041f5ab4c27671e0843"
Oct 30 21:45:06.807949 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:06.807925    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg_odh-ai-gateway-infra(c3712d27-9480-45ce-a089-caff86ddb3ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg" podUID="c3712d27-9480-45ce-a089-caff86ddb3ec"
Oct 30 21:45:06.809511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.809490    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6" event={"ID":"296ad3ea-d106-480c-b05c-2c44452d39ca","Type":"ContainerStarted","Data":"a3f8d14c2b2ae504ca50213928c1c6ad7bd18c2d0eaa7fca4de065267daed6ae"}
Oct 30 21:45:06.810090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.810073    4918 scope.go:122] "RemoveContainer" containerID="03c8cc09628a09ca107f16f27688b60db7d321c5742cfef265d8e3771a6778f0"
Oct 30 21:45:06.811527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.811507    4918 generic.go:356] "Generic (PLEG): container finished" podID="409b4a57-a05d-49c5-92e0-58a4b1712afc" containerID="375c23e52f2a61533475d58acba8368845d69f773a1c49b21d63cdcaff99db2e" exitCode=60
Oct 30 21:45:06.811604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.811587    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42" event={"ID":"409b4a57-a05d-49c5-92e0-58a4b1712afc","Type":"ContainerDied","Data":"375c23e52f2a61533475d58acba8368845d69f773a1c49b21d63cdcaff99db2e"}
Oct 30 21:45:06.811794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.811780    4918 scope.go:122] "RemoveContainer" containerID="375c23e52f2a61533475d58acba8368845d69f773a1c49b21d63cdcaff99db2e"
Oct 30 21:45:06.812021 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:06.812003    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42_odh-ai-gateway-infra(409b4a57-a05d-49c5-92e0-58a4b1712afc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc"
Oct 30 21:45:06.813261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.813243    4918 generic.go:356] "Generic (PLEG): container finished" podID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" containerID="0fedd7fe84b435bc2fd23097e044ff9734520d2fcf23877b73c04de7d8d90096" exitCode=60
Oct 30 21:45:06.813344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.813324    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk" event={"ID":"ee00d03a-36a1-47fe-be99-0cf595abd3a5","Type":"ContainerDied","Data":"0fedd7fe84b435bc2fd23097e044ff9734520d2fcf23877b73c04de7d8d90096"}
Oct 30 21:45:06.813587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.813571    4918 scope.go:122] "RemoveContainer" containerID="0fedd7fe84b435bc2fd23097e044ff9734520d2fcf23877b73c04de7d8d90096"
Oct 30 21:45:06.813863 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:06.813841    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk_odh-ai-gateway-infra(ee00d03a-36a1-47fe-be99-0cf595abd3a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5"
Oct 30 21:45:06.815865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.815843    4918 generic.go:356] "Generic (PLEG): container finished" podID="6bba61f8-4422-4206-8723-40d9ac86a4c2" containerID="672ec0326efe71d27102dc1a3b7afaf244b7d257be9224eb0d29734937f976e8" exitCode=60
Oct 30 21:45:06.815953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.815920    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7" event={"ID":"6bba61f8-4422-4206-8723-40d9ac86a4c2","Type":"ContainerDied","Data":"672ec0326efe71d27102dc1a3b7afaf244b7d257be9224eb0d29734937f976e8"}
Oct 30 21:45:06.816131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.816119    4918 scope.go:122] "RemoveContainer" containerID="672ec0326efe71d27102dc1a3b7afaf244b7d257be9224eb0d29734937f976e8"
Oct 30 21:45:06.816317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.816302    4918 scope.go:122] "RemoveContainer" containerID="7fd3c1973498c040ec81d0e2f3f8230fa043192ce6f3e458b6d0321e379b6e40"
Oct 30 21:45:06.816383 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:06.816368    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7_odh-ai-gateway-infra(6bba61f8-4422-4206-8723-40d9ac86a4c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2"
Oct 30 21:45:06.816629 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:06.816608    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z_odh-ai-gateway-infra(79a3898f-6fd6-4db6-811b-48fdef8f81ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba"
Oct 30 21:45:06.826614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:06.826596    4918 scope.go:122] "RemoveContainer" containerID="bae53ac8738dd2bf55cd472f9c6d19fb005f3cc452c639402f27ecfd797d5037"
Oct 30 21:45:07.217743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.217716    4918 scope.go:122] "RemoveContainer" containerID="4a9b3203b0e3b030779a006eaeeaba9f04ebd84f8f30d308050b0b0db21e0a19"
Oct 30 21:45:07.259328 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.259270    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6" podStartSLOduration=7.2592527239999995 podStartE2EDuration="7.259252724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:07.199834619 +0000 UTC m=+2620.246393766" watchObservedRunningTime="2026-10-30 21:45:07.259252724 +0000 UTC m=+2620.305811870"
Oct 30 21:45:07.381119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.381054    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv" podUID="f29441da-316c-4a9c-9fda-db62b0ff8cc9" containerName="payload-processing" containerID="cri-o://4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b" gracePeriod=25
Oct 30 21:45:07.517593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.515940    4918 scope.go:122] "RemoveContainer" containerID="b0bdcf9a7e00b0ca3319a9ebe87c7f5a5c49844197d2588eb701afde8589d735"
Oct 30 21:45:07.535466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.535436    4918 scope.go:122] "RemoveContainer" containerID="ba61cd5a2340ec201bec7c05bd1431a13a316d0245714bab4683cb4050c10943"
Oct 30 21:45:07.554467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.554438    4918 scope.go:122] "RemoveContainer" containerID="8152e08d518a530ca604ef428873d19a054f8ab14c3473425dd4e88d0b065a43"
Oct 30 21:45:07.571850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.571830    4918 scope.go:122] "RemoveContainer" containerID="ae56af92599b4daad04d32f1ba590f2b66a5716f271566c6361b8c6e391ae0d8"
Oct 30 21:45:07.610014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.609308    4918 scope.go:122] "RemoveContainer" containerID="f26ac3fe35f146da760e0900de8035eac581d660b803e9bdb39beaab089c9996"
Oct 30 21:45:07.692982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.692947    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:45:07.794239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.794202    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-kube-api-access-28lk8\" (UniqueName: \"kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-kube-api-access-28lk8\") pod \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") "
Oct 30 21:45:07.794881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.794367    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-routing-overlay-volume\") pod \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") "
Oct 30 21:45:07.794881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.794524    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-provider-credentials\") pod \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") "
Oct 30 21:45:07.794881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.794721    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-plugins-config-volume\") pod \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\" (UID: \"f29441da-316c-4a9c-9fda-db62b0ff8cc9\") "
Oct 30 21:45:07.796559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.796531    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-routing-overlay-volume" pod "f29441da-316c-4a9c-9fda-db62b0ff8cc9" (UID: "f29441da-316c-4a9c-9fda-db62b0ff8cc9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:07.796768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.796746    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-plugins-config-volume" pod "f29441da-316c-4a9c-9fda-db62b0ff8cc9" (UID: "f29441da-316c-4a9c-9fda-db62b0ff8cc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:07.799017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.798990    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-kube-api-access-28lk8" pod "f29441da-316c-4a9c-9fda-db62b0ff8cc9" (UID: "f29441da-316c-4a9c-9fda-db62b0ff8cc9"). InnerVolumeSpecName "kube-api-access-28lk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:07.799522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.799493    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-provider-credentials" pod "f29441da-316c-4a9c-9fda-db62b0ff8cc9" (UID: "f29441da-316c-4a9c-9fda-db62b0ff8cc9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:07.801548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.801522    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"]
Oct 30 21:45:07.802588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.802571    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f29441da-316c-4a9c-9fda-db62b0ff8cc9" containerName="payload-processing"
Oct 30 21:45:07.802639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.802590    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29441da-316c-4a9c-9fda-db62b0ff8cc9" containerName="payload-processing"
Oct 30 21:45:07.802785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.802775    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29441da-316c-4a9c-9fda-db62b0ff8cc9" containerName="payload-processing"
Oct 30 21:45:07.807738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.807719    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:07.826946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.826923    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-d4a0a5-openshift-default-dockercfg-8sgkb\""
Oct 30 21:45:07.831089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.831061    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"]
Oct 30 21:45:07.837274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.837250    4918 scope.go:122] "RemoveContainer" containerID="8db3775c6566b63512f2f53c9e28e49f253da12ebd228d469960fd932ddf95c4"
Oct 30 21:45:07.837783 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:07.837761    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6_odh-ai-gateway-infra(83727964-4998-42ed-82b2-7ac611269201)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6" podUID="83727964-4998-42ed-82b2-7ac611269201"
Oct 30 21:45:07.841561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.841539    4918 scope.go:122] "RemoveContainer" containerID="5a0558999cf6e2c2eead69207f38e48a9dad7d1abc3fec82969047ef3c3c1d61"
Oct 30 21:45:07.841953 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:07.841926    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b_odh-ai-gateway-infra(e65d8a21-107a-40ad-b8fc-967cb5008270)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b" podUID="e65d8a21-107a-40ad-b8fc-967cb5008270"
Oct 30 21:45:07.842548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.842531    4918 scope.go:122] "RemoveContainer" containerID="6dd25e3e03dce0854383c9fe95c1d35112a07e2541811710850b4c6bdfd84232"
Oct 30 21:45:07.843060 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:07.843033    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx_odh-ai-gateway-infra(5a319997-3bb4-419a-a27b-23849ee7691b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx" podUID="5a319997-3bb4-419a-a27b-23849ee7691b"
Oct 30 21:45:07.846724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.846697    4918 generic.go:356] "Generic (PLEG): container finished" podID="f6b82123-e7fd-41ff-9655-c2610e1973bf" containerID="15a956bc024235f56ecbfdef23b6848a75252a7c1dc9738c290e14cbd94ed0a9" exitCode=60
Oct 30 21:45:07.846821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.846796    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j" event={"ID":"f6b82123-e7fd-41ff-9655-c2610e1973bf","Type":"ContainerDied","Data":"15a956bc024235f56ecbfdef23b6848a75252a7c1dc9738c290e14cbd94ed0a9"}
Oct 30 21:45:07.847065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.846844    4918 scope.go:122] "RemoveContainer" containerID="e1444b740da819b032cf43dee3f8e8de8110a35e6e0c902ce85237960492c2ec"
Oct 30 21:45:07.847664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.847645    4918 scope.go:122] "RemoveContainer" containerID="15a956bc024235f56ecbfdef23b6848a75252a7c1dc9738c290e14cbd94ed0a9"
Oct 30 21:45:07.848120 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:07.848092    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j_odh-ai-gateway-infra(f6b82123-e7fd-41ff-9655-c2610e1973bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j" podUID="f6b82123-e7fd-41ff-9655-c2610e1973bf"
Oct 30 21:45:07.858051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.858016    4918 generic.go:356] "Generic (PLEG): container finished" podID="06cfd24a-7880-40a4-98ed-0c9326491795" containerID="0679ac3a391382a406b51965d332fc1d35113e00567b36fd783b42a279a378a1" exitCode=60
Oct 30 21:45:07.858171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.858082    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w" event={"ID":"06cfd24a-7880-40a4-98ed-0c9326491795","Type":"ContainerDied","Data":"0679ac3a391382a406b51965d332fc1d35113e00567b36fd783b42a279a378a1"}
Oct 30 21:45:07.858377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.858361    4918 scope.go:122] "RemoveContainer" containerID="0679ac3a391382a406b51965d332fc1d35113e00567b36fd783b42a279a378a1"
Oct 30 21:45:07.858724 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:07.858700    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w_odh-ai-gateway-infra(06cfd24a-7880-40a4-98ed-0c9326491795)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w" podUID="06cfd24a-7880-40a4-98ed-0c9326491795"
Oct 30 21:45:07.860543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.860494    4918 generic.go:356] "Generic (PLEG): container finished" podID="f29441da-316c-4a9c-9fda-db62b0ff8cc9" containerID="4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b" exitCode=0
Oct 30 21:45:07.860696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.860556    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"
Oct 30 21:45:07.861164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.860559    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv" event={"ID":"f29441da-316c-4a9c-9fda-db62b0ff8cc9","Type":"ContainerDied","Data":"4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b"}
Oct 30 21:45:07.861356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.861333    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv" event={"ID":"f29441da-316c-4a9c-9fda-db62b0ff8cc9","Type":"ContainerDied","Data":"9cac2ac9a2201a921b18295e7126d4890c3a3ca702091fc5b95ecf6f1217f1c9"}
Oct 30 21:45:07.865601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.865573    4918 scope.go:122] "RemoveContainer" containerID="8cfa7a50db8b7e393a04285f412301ccb3892dbc1beff4050d49c0481c0b062c"
Oct 30 21:45:07.865893 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:07.865859    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs_odh-ai-gateway-infra(e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af"
Oct 30 21:45:07.867748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.867725    4918 generic.go:356] "Generic (PLEG): container finished" podID="2c42b67a-e483-4511-9be1-005d53f02bdc" containerID="66db99a02ad0afbfc774e06d30c398595426ab1cf161c7c2bc7f64b0e0a37fa0" exitCode=60
Oct 30 21:45:07.867844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.867804    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd" event={"ID":"2c42b67a-e483-4511-9be1-005d53f02bdc","Type":"ContainerDied","Data":"66db99a02ad0afbfc774e06d30c398595426ab1cf161c7c2bc7f64b0e0a37fa0"}
Oct 30 21:45:07.868166 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.868135    4918 scope.go:122] "RemoveContainer" containerID="66db99a02ad0afbfc774e06d30c398595426ab1cf161c7c2bc7f64b0e0a37fa0"
Oct 30 21:45:07.868509 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:07.868488    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd_odh-ai-gateway-infra(2c42b67a-e483-4511-9be1-005d53f02bdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd" podUID="2c42b67a-e483-4511-9be1-005d53f02bdc"
Oct 30 21:45:07.877579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.877360    4918 scope.go:122] "RemoveContainer" containerID="b83481189ef09bda65f1973b330e1405df7fe27e45d4fdbd82279bd35904d936"
Oct 30 21:45:07.877896 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:07.877876    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj_odh-ai-gateway-infra(4b80735b-3681-4774-b4a3-821f37748327)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj" podUID="4b80735b-3681-4774-b4a3-821f37748327"
Oct 30 21:45:07.879693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.879674    4918 generic.go:356] "Generic (PLEG): container finished" podID="296ad3ea-d106-480c-b05c-2c44452d39ca" containerID="a3f8d14c2b2ae504ca50213928c1c6ad7bd18c2d0eaa7fca4de065267daed6ae" exitCode=60
Oct 30 21:45:07.879782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.879770    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6" event={"ID":"296ad3ea-d106-480c-b05c-2c44452d39ca","Type":"ContainerDied","Data":"a3f8d14c2b2ae504ca50213928c1c6ad7bd18c2d0eaa7fca4de065267daed6ae"}
Oct 30 21:45:07.880087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.880070    4918 scope.go:122] "RemoveContainer" containerID="a3f8d14c2b2ae504ca50213928c1c6ad7bd18c2d0eaa7fca4de065267daed6ae"
Oct 30 21:45:07.880357 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:07.880331    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6_odh-ai-gateway-infra(296ad3ea-d106-480c-b05c-2c44452d39ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6" podUID="296ad3ea-d106-480c-b05c-2c44452d39ca"
Oct 30 21:45:07.881462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.880350    4918 scope.go:122] "RemoveContainer" containerID="062f439824833622dd591bdfcd28fcc2986def41ce5ffb55df50075f39e81981"
Oct 30 21:45:07.882493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.882474    4918 scope.go:122] "RemoveContainer" containerID="375c23e52f2a61533475d58acba8368845d69f773a1c49b21d63cdcaff99db2e"
Oct 30 21:45:07.882732 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:07.882710    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42_odh-ai-gateway-infra(409b4a57-a05d-49c5-92e0-58a4b1712afc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc"
Oct 30 21:45:07.887082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.887052    4918 scope.go:122] "RemoveContainer" containerID="0fedd7fe84b435bc2fd23097e044ff9734520d2fcf23877b73c04de7d8d90096"
Oct 30 21:45:07.906255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.906216    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp5sh\" (UniqueName: \"kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-kube-api-access-qp5sh\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:07.906559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.906540    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istiod-ca-cert\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:07.922845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.917598    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-socket\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:07.938759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.938721    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-credential-socket\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:07.960441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.950347    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-envoy\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:07.960987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.960935    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-data\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:07.964734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.963897    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-podinfo\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:07.964734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.963962    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-token\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:07.964734 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:07.889356    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk_odh-ai-gateway-infra(ee00d03a-36a1-47fe-be99-0cf595abd3a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5"
Oct 30 21:45:07.964734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.964322    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-certs\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:07.964734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.964530    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28lk8\" (UniqueName: \"kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-kube-api-access-28lk8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:07.964734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.964551    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:07.964734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.964567    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f29441da-316c-4a9c-9fda-db62b0ff8cc9-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:07.964734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:07.964582    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f29441da-316c-4a9c-9fda-db62b0ff8cc9-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:08.004172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.004107    4918 scope.go:122] "RemoveContainer" containerID="672ec0326efe71d27102dc1a3b7afaf244b7d257be9224eb0d29734937f976e8"
Oct 30 21:45:08.004656 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:08.004628    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7_odh-ai-gateway-infra(6bba61f8-4422-4206-8723-40d9ac86a4c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2"
Oct 30 21:45:08.013005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.012980    4918 scope.go:122] "RemoveContainer" containerID="1300cc02b07c713f550535f2dde2ca35a083e9fb7302e7ba0e5420ff6000d611"
Oct 30 21:45:08.015048 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:08.015020    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp_odh-ai-gateway-infra(aedc6368-2835-4821-bf6c-9cb838f21611)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp" podUID="aedc6368-2835-4821-bf6c-9cb838f21611"
Oct 30 21:45:08.020874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.020846    4918 generic.go:356] "Generic (PLEG): container finished" podID="a60f039f-582f-4178-916d-5bf0260ed803" containerID="d7943d39b1c8c61643c668cb64ceaebbfa2894011129d254b6daa4a21a2353bd" exitCode=60
Oct 30 21:45:08.021117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.020981    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6" event={"ID":"a60f039f-582f-4178-916d-5bf0260ed803","Type":"ContainerDied","Data":"d7943d39b1c8c61643c668cb64ceaebbfa2894011129d254b6daa4a21a2353bd"}
Oct 30 21:45:08.021481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.021384    4918 scope.go:122] "RemoveContainer" containerID="d7943d39b1c8c61643c668cb64ceaebbfa2894011129d254b6daa4a21a2353bd"
Oct 30 21:45:08.021943 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:08.021919    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6_odh-ai-gateway-infra(a60f039f-582f-4178-916d-5bf0260ed803)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6" podUID="a60f039f-582f-4178-916d-5bf0260ed803"
Oct 30 21:45:08.034002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.033981    4918 scope.go:122] "RemoveContainer" containerID="4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b"
Oct 30 21:45:08.053027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.052427    4918 scope.go:122] "RemoveContainer" containerID="4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b"
Oct 30 21:45:08.053638 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:08.053606    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b\": container with ID starting with 4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b not found: ID does not exist" containerID="4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b"
Oct 30 21:45:08.053747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.053648    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b"} err="failed to get container status \"4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b\": rpc error: code = NotFound desc = could not find container \"4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b\": container with ID starting with 4d566f4fe322a9a5538e0c9344b5c7c72cf81391414b63ee56a358c4feb7099b not found: ID does not exist"
Oct 30 21:45:08.053747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.053671    4918 scope.go:122] "RemoveContainer" containerID="42c70031e93d8e5946ca4d99d4b2ecbc0fb19fddba99125cd3ad7c80e9da2426"
Oct 30 21:45:08.066097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.066071    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp5sh\" (UniqueName: \"kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-kube-api-access-qp5sh\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.066446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.066371    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istiod-ca-cert\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.066841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.066787    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-socket\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.067147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.067119    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-socket\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.067266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.067243    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-credential-socket\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.067349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.067282    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-envoy\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.067466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.067406    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istiod-ca-cert\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.067466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.067437    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-data\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.067595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.067494    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-podinfo\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.067595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.067547    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-token\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.067750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.067710    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-certs\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.067832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.067805    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-data\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.068109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.068084    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-certs\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.068210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.068142    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-credential-socket\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.070594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.070568    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-envoy\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.070786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.070767    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-podinfo\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.078256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.078236    4918 scope.go:122] "RemoveContainer" containerID="fe9180e53514257dbbfd9536c59b01f848b8769ef8fde68ca74a40de717eeb33"
Oct 30 21:45:08.099883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.099334    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-token\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.099883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.099838    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp5sh\" (UniqueName: \"kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-kube-api-access-qp5sh\") pod \"e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") " pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.141314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.141288    4918 scope.go:122] "RemoveContainer" containerID="76a5131da70d8e9d758c731d7414cda2895fd02b0d288323914181482da8ee83"
Oct 30 21:45:08.177206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.177179    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:08.252161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.252132    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"]
Oct 30 21:45:08.264182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.264152    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-86f5c55f49-nxblv"]
Oct 30 21:45:08.488312 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:45:08.488278    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4378181_49fc_4f6b_b9ca_6700bec0fbc8.slice/crio-8774cf624ba44418240277953497a4a3619323e5a52e3a5d8734e185f003464f WatchSource:0}: Error finding container 8774cf624ba44418240277953497a4a3619323e5a52e3a5d8734e185f003464f: Status 404 returned error can't find the container with id 8774cf624ba44418240277953497a4a3619323e5a52e3a5d8734e185f003464f
Oct 30 21:45:08.490170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.490143    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"]
Oct 30 21:45:08.491543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.491515    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:45:08.491622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.491588    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:45:08.491660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:08.491627    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:45:09.042575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:09.042537    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" event={"ID":"f4378181-49fc-4f6b-b9ca-6700bec0fbc8","Type":"ContainerStarted","Data":"ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af"}
Oct 30 21:45:09.043090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:09.042630    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" event={"ID":"f4378181-49fc-4f6b-b9ca-6700bec0fbc8","Type":"ContainerStarted","Data":"8774cf624ba44418240277953497a4a3619323e5a52e3a5d8734e185f003464f"}
Oct 30 21:45:09.044868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:09.044837    4918 scope.go:122] "RemoveContainer" containerID="d7943d39b1c8c61643c668cb64ceaebbfa2894011129d254b6daa4a21a2353bd"
Oct 30 21:45:09.045440 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:09.045372    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6_odh-ai-gateway-infra(a60f039f-582f-4178-916d-5bf0260ed803)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6" podUID="a60f039f-582f-4178-916d-5bf0260ed803"
Oct 30 21:45:09.049084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:09.049051    4918 scope.go:122] "RemoveContainer" containerID="0679ac3a391382a406b51965d332fc1d35113e00567b36fd783b42a279a378a1"
Oct 30 21:45:09.050430 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:09.049469    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w_odh-ai-gateway-infra(06cfd24a-7880-40a4-98ed-0c9326491795)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w" podUID="06cfd24a-7880-40a4-98ed-0c9326491795"
Oct 30 21:45:09.083277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:09.083215    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" podStartSLOduration=2.083200664 podStartE2EDuration="2.083200664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:09.081503304 +0000 UTC m=+2622.128062469" watchObservedRunningTime="2026-10-30 21:45:09.083200664 +0000 UTC m=+2622.129759810"
Oct 30 21:45:09.083705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:09.083680    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29441da-316c-4a9c-9fda-db62b0ff8cc9" path="/var/lib/kubelet/pods/f29441da-316c-4a9c-9fda-db62b0ff8cc9/volumes"
Oct 30 21:45:09.178364 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:09.178330    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:09.180323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:09.180298    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 21:45:09.180457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:09.180344    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" podUID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 21:45:09.524470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:09.524427    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs" podUID="2f5bfd22-d58e-471a-aef3-565da8291106" containerName="payload-processing" containerID="cri-o://9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172" gracePeriod=25
Oct 30 21:45:09.684509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:09.684481    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:45:10.047973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.047938    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:45:10.063122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.063091    4918 generic.go:356] "Generic (PLEG): container finished" podID="2f5bfd22-d58e-471a-aef3-565da8291106" containerID="9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172" exitCode=0
Oct 30 21:45:10.063287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.063131    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"
Oct 30 21:45:10.063287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.063168    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs" event={"ID":"2f5bfd22-d58e-471a-aef3-565da8291106","Type":"ContainerDied","Data":"9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172"}
Oct 30 21:45:10.063287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.063225    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs" event={"ID":"2f5bfd22-d58e-471a-aef3-565da8291106","Type":"ContainerDied","Data":"b477349525b4adab0f010f5c2e58483f0228de6b835fdc2a443b265b6eeb2340"}
Oct 30 21:45:10.063287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.063246    4918 scope.go:122] "RemoveContainer" containerID="9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172"
Oct 30 21:45:10.077441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.077404    4918 scope.go:122] "RemoveContainer" containerID="9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172"
Oct 30 21:45:10.077790 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:10.077770    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172\": container with ID starting with 9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172 not found: ID does not exist" containerID="9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172"
Oct 30 21:45:10.077876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.077799    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172"} err="failed to get container status \"9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172\": rpc error: code = NotFound desc = could not find container \"9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172\": container with ID starting with 9ea8512f6236c31067b9c78d9630fc250869061741117abeb31dbc22f5b3d172 not found: ID does not exist"
Oct 30 21:45:10.177879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.177844    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 21:45:10.178059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.177902    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" podUID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 21:45:10.215168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.215139    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-plugins-config-volume\") pod \"2f5bfd22-d58e-471a-aef3-565da8291106\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") "
Oct 30 21:45:10.215342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.215193    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-kube-api-access-j6m9p\" (UniqueName: \"kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-kube-api-access-j6m9p\") pod \"2f5bfd22-d58e-471a-aef3-565da8291106\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") "
Oct 30 21:45:10.215459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.215364    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-provider-credentials\") pod \"2f5bfd22-d58e-471a-aef3-565da8291106\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") "
Oct 30 21:45:10.215640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.215612    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-plugins-config-volume" pod "2f5bfd22-d58e-471a-aef3-565da8291106" (UID: "2f5bfd22-d58e-471a-aef3-565da8291106"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:10.215823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.215804    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-routing-overlay-volume\") pod \"2f5bfd22-d58e-471a-aef3-565da8291106\" (UID: \"2f5bfd22-d58e-471a-aef3-565da8291106\") "
Oct 30 21:45:10.216213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.216193    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-routing-overlay-volume" pod "2f5bfd22-d58e-471a-aef3-565da8291106" (UID: "2f5bfd22-d58e-471a-aef3-565da8291106"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:10.217028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.217007    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:10.217103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.217035    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f5bfd22-d58e-471a-aef3-565da8291106-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:10.220456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.218341    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-kube-api-access-j6m9p" pod "2f5bfd22-d58e-471a-aef3-565da8291106" (UID: "2f5bfd22-d58e-471a-aef3-565da8291106"). InnerVolumeSpecName "kube-api-access-j6m9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:10.220456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.218449    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-provider-credentials" pod "2f5bfd22-d58e-471a-aef3-565da8291106" (UID: "2f5bfd22-d58e-471a-aef3-565da8291106"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:10.319044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.319000    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6m9p\" (UniqueName: \"kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-kube-api-access-j6m9p\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:10.319044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.319037    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f5bfd22-d58e-471a-aef3-565da8291106-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:10.417893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.417861    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"]
Oct 30 21:45:10.423309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.423278    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e596df/payload-processing-external-model-e2e-extmod-a-e596df-84fc9b6cs"]
Oct 30 21:45:10.742566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:10.742527    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c" podUID="8093b0c3-59d2-4c75-8b63-286b5ccaf1cf" containerName="payload-processing" containerID="cri-o://52d6bf9175a21b874c1e002307ec178db9b4da4dec117f2fe354712f00ec9125" gracePeriod=25
Oct 30 21:45:11.083090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.083047    4918 generic.go:356] "Generic (PLEG): container finished" podID="8093b0c3-59d2-4c75-8b63-286b5ccaf1cf" containerID="52d6bf9175a21b874c1e002307ec178db9b4da4dec117f2fe354712f00ec9125" exitCode=0
Oct 30 21:45:11.085711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.085682    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5bfd22-d58e-471a-aef3-565da8291106" path="/var/lib/kubelet/pods/2f5bfd22-d58e-471a-aef3-565da8291106/volumes"
Oct 30 21:45:11.086172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.086147    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c" event={"ID":"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf","Type":"ContainerDied","Data":"52d6bf9175a21b874c1e002307ec178db9b4da4dec117f2fe354712f00ec9125"}
Oct 30 21:45:11.086311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.086199    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c" event={"ID":"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf","Type":"ContainerDied","Data":"06ee8773ee67bb9873d2c885f9cdc45b8ba25bd5357a2dff7793199d6e201483"}
Oct 30 21:45:11.086311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.086226    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ee8773ee67bb9873d2c885f9cdc45b8ba25bd5357a2dff7793199d6e201483"
Oct 30 21:45:11.124840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.124816    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:45:11.177890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.177857    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 21:45:11.178050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.177908    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" podUID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 21:45:11.244914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.244878    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-provider-credentials\") pod \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") "
Oct 30 21:45:11.245162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.245085    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-plugins-config-volume\") pod \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") "
Oct 30 21:45:11.245444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.245369    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-routing-overlay-volume\") pod \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") "
Oct 30 21:45:11.245606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.245585    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-kube-api-access-4ncng\" (UniqueName: \"kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-kube-api-access-4ncng\") pod \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\" (UID: \"8093b0c3-59d2-4c75-8b63-286b5ccaf1cf\") "
Oct 30 21:45:11.246315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.245585    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-plugins-config-volume" pod "8093b0c3-59d2-4c75-8b63-286b5ccaf1cf" (UID: "8093b0c3-59d2-4c75-8b63-286b5ccaf1cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:11.246752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.245958    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-routing-overlay-volume" pod "8093b0c3-59d2-4c75-8b63-286b5ccaf1cf" (UID: "8093b0c3-59d2-4c75-8b63-286b5ccaf1cf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:11.247042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.247004    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:11.247042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.247032    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:11.247863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.247842    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-provider-credentials" pod "8093b0c3-59d2-4c75-8b63-286b5ccaf1cf" (UID: "8093b0c3-59d2-4c75-8b63-286b5ccaf1cf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:11.249764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.249744    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-kube-api-access-4ncng" pod "8093b0c3-59d2-4c75-8b63-286b5ccaf1cf" (UID: "8093b0c3-59d2-4c75-8b63-286b5ccaf1cf"). InnerVolumeSpecName "kube-api-access-4ncng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:11.348908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.348827    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:11.348908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:11.348857    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ncng\" (UniqueName: \"kubernetes.io/projected/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf-kube-api-access-4ncng\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:12.116982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.116946    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"
Oct 30 21:45:12.144843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.144813    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"]
Oct 30 21:45:12.145778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.145763    4918 cpu_manager.go:414] "containerMap: removing container" podUID="8093b0c3-59d2-4c75-8b63-286b5ccaf1cf" containerName="payload-processing"
Oct 30 21:45:12.145838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.145779    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8093b0c3-59d2-4c75-8b63-286b5ccaf1cf" containerName="payload-processing"
Oct 30 21:45:12.145838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.145794    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2f5bfd22-d58e-471a-aef3-565da8291106" containerName="payload-processing"
Oct 30 21:45:12.145838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.145799    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5bfd22-d58e-471a-aef3-565da8291106" containerName="payload-processing"
Oct 30 21:45:12.145998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.145987    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8093b0c3-59d2-4c75-8b63-286b5ccaf1cf" containerName="payload-processing"
Oct 30 21:45:12.146032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.146003    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5bfd22-d58e-471a-aef3-565da8291106" containerName="payload-processing"
Oct 30 21:45:12.151380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.151358    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.156168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.156146    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"]
Oct 30 21:45:12.161664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.161643    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1ed97/payload-processing-external-model-e2e-extmod-b-f1ed97-84cbs5d9c"]
Oct 30 21:45:12.175522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.175496    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"]
Oct 30 21:45:12.177697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.177675    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 21:45:12.177793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.177730    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" podUID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 21:45:12.205416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.194347    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f9e5a7-openshift-default-dockercfg-7hvlf\""
Oct 30 21:45:12.268859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.268821    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-data\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.269117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.269084    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-podinfo\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.269262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.269136    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-certs\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.269262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.269246    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istiod-ca-cert\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.269454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.269337    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-credential-socket\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.269524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.269469    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-token\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.269602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.269582    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-socket\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.269661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.269640    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzsc\" (UniqueName: \"kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-kube-api-access-zbzsc\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.269827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.269792    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-envoy\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.372820    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-data\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.372872    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-podinfo\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.372899    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-certs\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.372952    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istiod-ca-cert\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.373025    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-credential-socket\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.373124    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-token\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.373211    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-socket\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.373263    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzsc\" (UniqueName: \"kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-kube-api-access-zbzsc\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.373378    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-envoy\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.377343    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-envoy\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.377778    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-data\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.380694    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-credential-socket\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.381723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.381267    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-certs\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.411747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.382013    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istiod-ca-cert\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.411747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.382253    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-socket\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.411747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.383176    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-podinfo\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.411747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.410292    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-token\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.451558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.451525    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzsc\" (UniqueName: \"kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-kube-api-access-zbzsc\") pod \"e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") " pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.467917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.467892    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:12.818260 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:45:12.818226    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddd4fc4_2a3a_450b_898b_81d92045fc65.slice/crio-a701267df416a01ccbfd74a8fb0d344a492150065cb0e9a6c05c35edb7a9ea02 WatchSource:0}: Error finding container a701267df416a01ccbfd74a8fb0d344a492150065cb0e9a6c05c35edb7a9ea02: Status 404 returned error can't find the container with id a701267df416a01ccbfd74a8fb0d344a492150065cb0e9a6c05c35edb7a9ea02
Oct 30 21:45:12.818888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.818865    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"]
Oct 30 21:45:12.821479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.821451    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:45:12.821546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.821514    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:45:12.821598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.821544    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:45:12.865090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.865066    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"]
Oct 30 21:45:12.872487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.872468    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:12.895446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.883633    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-b30a42-openshift-default-dockercfg-qgz6d\""
Oct 30 21:45:12.907996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.902505    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"]
Oct 30 21:45:12.985160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.985123    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-credential-socket\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:12.985335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.985240    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431ab330-ad11-4272-9adf-1584a08e3aaf-istiod-ca-cert\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:12.985335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.985296    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-socket\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:12.985335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:12.985324    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-podinfo\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.020812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.013544    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-token\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.020812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.013661    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-certs\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.020812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.013765    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-envoy\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.020812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.013805    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-data\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.020812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.013883    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5wf4\" (UniqueName: \"kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-kube-api-access-j5wf4\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.082303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.082227    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8093b0c3-59d2-4c75-8b63-286b5ccaf1cf" path="/var/lib/kubelet/pods/8093b0c3-59d2-4c75-8b63-286b5ccaf1cf/volumes"
Oct 30 21:45:13.115211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.115164    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5wf4\" (UniqueName: \"kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-kube-api-access-j5wf4\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.115475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.115344    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-credential-socket\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.115581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.115497    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431ab330-ad11-4272-9adf-1584a08e3aaf-istiod-ca-cert\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.115581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.115555    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-socket\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.115685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.115583    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-podinfo\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.115737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.115715    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-token\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.115795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.115746    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-credential-socket\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.116042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.116018    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-socket\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.116147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.116064    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-certs\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.116147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.116069    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431ab330-ad11-4272-9adf-1584a08e3aaf-istiod-ca-cert\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.116297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.116253    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-envoy\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.116297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.116288    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-data\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.116469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.116336    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-certs\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.116633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.116614    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-data\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.118854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.118828    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-podinfo\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.119176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.118924    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-envoy\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.124062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.124044    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5wf4\" (UniqueName: \"kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-kube-api-access-j5wf4\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.124453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.124432    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-token\") pod \"e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") " pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.130435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.130382    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" event={"ID":"4ddd4fc4-2a3a-450b-898b-81d92045fc65","Type":"ContainerStarted","Data":"c0bd31ec03c9205b183fee8e9c2cb4d89534902413ffd1adae2b92ce6bb56e8c"}
Oct 30 21:45:13.130579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.130564    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" event={"ID":"4ddd4fc4-2a3a-450b-898b-81d92045fc65","Type":"ContainerStarted","Data":"a701267df416a01ccbfd74a8fb0d344a492150065cb0e9a6c05c35edb7a9ea02"}
Oct 30 21:45:13.163365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.163304    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" podStartSLOduration=1.163286426 podStartE2EDuration="1.163286426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:13.156546102 +0000 UTC m=+2626.203105249" watchObservedRunningTime="2026-10-30 21:45:13.163286426 +0000 UTC m=+2626.209845574"
Oct 30 21:45:13.177863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.177832    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 21:45:13.178000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.177889    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" podUID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 21:45:13.221099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.221054    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:13.468654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.468412    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:13.472549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.472345    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 30 21:45:13.472549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.472436    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" podUID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 30 21:45:13.734691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.733023    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"]
Oct 30 21:45:13.745853 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:45:13.745807    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431ab330_ad11_4272_9adf_1584a08e3aaf.slice/crio-42433ff58148a98acec40c89b8dbe1d1b4e8b6033f1db26931760b965a594fb9 WatchSource:0}: Error finding container 42433ff58148a98acec40c89b8dbe1d1b4e8b6033f1db26931760b965a594fb9: Status 404 returned error can't find the container with id 42433ff58148a98acec40c89b8dbe1d1b4e8b6033f1db26931760b965a594fb9
Oct 30 21:45:13.749995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.749862    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:45:13.749995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.749954    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:45:13.749995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:13.749995    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:45:14.167958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:14.167860    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" event={"ID":"431ab330-ad11-4272-9adf-1584a08e3aaf","Type":"ContainerStarted","Data":"874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8"}
Oct 30 21:45:14.168455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:14.167964    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" event={"ID":"431ab330-ad11-4272-9adf-1584a08e3aaf","Type":"ContainerStarted","Data":"42433ff58148a98acec40c89b8dbe1d1b4e8b6033f1db26931760b965a594fb9"}
Oct 30 21:45:14.178408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:14.178358    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused" start-of-body=
Oct 30 21:45:14.178567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:14.178428    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" podUID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.118:15021/healthz/ready\": dial tcp 10.132.1.118:15021: connect: connection refused"
Oct 30 21:45:14.221479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:14.214959    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" podStartSLOduration=2.214935765 podStartE2EDuration="2.214935765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:14.202865239 +0000 UTC m=+2627.249424387" watchObservedRunningTime="2026-10-30 21:45:14.214935765 +0000 UTC m=+2627.261494910"
Oct 30 21:45:14.227425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:14.221897    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:14.239413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:14.231541    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 30 21:45:14.239413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:14.231597    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 30 21:45:14.473512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:14.468963    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 30 21:45:14.473512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:14.469032    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" podUID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 30 21:45:15.214451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:15.211769    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:15.236458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:15.230458    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:15.245410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:15.242145    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 30 21:45:15.245410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:15.242196    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 30 21:45:15.253911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:15.253734    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:45:15.534749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:15.526021    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 30 21:45:15.534749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:15.526109    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" podUID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 30 21:45:16.245490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:16.245455    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 30 21:45:16.246020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:16.245513    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 30 21:45:16.470138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:16.470105    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 30 21:45:16.470292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:16.470156    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" podUID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 30 21:45:17.175992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:17.175965    4918 scope.go:122] "RemoveContainer" containerID="5a0558999cf6e2c2eead69207f38e48a9dad7d1abc3fec82969047ef3c3c1d61"
Oct 30 21:45:17.176459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:17.176384    4918 scope.go:122] "RemoveContainer" containerID="7fd3c1973498c040ec81d0e2f3f8230fa043192ce6f3e458b6d0321e379b6e40"
Oct 30 21:45:17.176630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:17.176611    4918 scope.go:122] "RemoveContainer" containerID="e65069a6f0c0a70ff185401140b9f1ea9c6cc6d677d45041f5ab4c27671e0843"
Oct 30 21:45:17.243502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:17.241277    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 30 21:45:17.243502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:17.241331    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 30 21:45:17.469215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:17.469188    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 30 21:45:17.469585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:17.469234    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" podUID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 30 21:45:18.106483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.082518    4918 scope.go:122] "RemoveContainer" containerID="672ec0326efe71d27102dc1a3b7afaf244b7d257be9224eb0d29734937f976e8"
Oct 30 21:45:18.106483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.083780    4918 scope.go:122] "RemoveContainer" containerID="0fedd7fe84b435bc2fd23097e044ff9734520d2fcf23877b73c04de7d8d90096"
Oct 30 21:45:18.106483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.084732    4918 scope.go:122] "RemoveContainer" containerID="8db3775c6566b63512f2f53c9e28e49f253da12ebd228d469960fd932ddf95c4"
Oct 30 21:45:18.160639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.151487    4918 scope.go:122] "RemoveContainer" containerID="1300cc02b07c713f550535f2dde2ca35a083e9fb7302e7ba0e5420ff6000d611"
Oct 30 21:45:18.160639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.151748    4918 scope.go:122] "RemoveContainer" containerID="d7943d39b1c8c61643c668cb64ceaebbfa2894011129d254b6daa4a21a2353bd"
Oct 30 21:45:18.260480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.259835    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 30 21:45:18.260480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.259930    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 30 21:45:18.284088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.284053    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z" event={"ID":"79a3898f-6fd6-4db6-811b-48fdef8f81ba","Type":"ContainerStarted","Data":"b161fef17a17b3f6d762468832aa2be10ffb6559b6327816895adfa73f9c6d5a"}
Oct 30 21:45:18.303727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.303690    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b" event={"ID":"e65d8a21-107a-40ad-b8fc-967cb5008270","Type":"ContainerStarted","Data":"b9d52ca549490fb209f6dad63f269e9a042c855177af78fa3cf6b81a9cf94e73"}
Oct 30 21:45:18.349750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.349707    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg" event={"ID":"c3712d27-9480-45ce-a089-caff86ddb3ec","Type":"ContainerStarted","Data":"a05c70cb41268942073d86e7d37dcc7d2e49234b60b0cfaed364ffc3857840dc"}
Oct 30 21:45:18.358479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.352878    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z" podStartSLOduration=18.35286013 podStartE2EDuration="18.35286013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:18.334891525 +0000 UTC m=+2631.381450695" watchObservedRunningTime="2026-10-30 21:45:18.35286013 +0000 UTC m=+2631.399419277"
Oct 30 21:45:18.468908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.468881    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 30 21:45:18.469055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:18.468935    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" podUID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 30 21:45:19.077630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.077605    4918 scope.go:122] "RemoveContainer" containerID="375c23e52f2a61533475d58acba8368845d69f773a1c49b21d63cdcaff99db2e"
Oct 30 21:45:19.078433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.078404    4918 scope.go:122] "RemoveContainer" containerID="6dd25e3e03dce0854383c9fe95c1d35112a07e2541811710850b4c6bdfd84232"
Oct 30 21:45:19.078748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.078728    4918 scope.go:122] "RemoveContainer" containerID="e4e09214c9a7354ae230fee9ce6d438787a3d451188d1a388e24135ac8685940"
Oct 30 21:45:19.344414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.341541    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 30 21:45:19.344414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.341607    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 30 21:45:19.483543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.483508    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"]
Oct 30 21:45:19.484421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.483704    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Oct 30 21:45:19.484655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.484630    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" podUID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Oct 30 21:45:19.553418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.547937    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"]
Oct 30 21:45:19.558480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.558451    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"]
Oct 30 21:45:19.569494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.566554    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp" event={"ID":"aedc6368-2835-4821-bf6c-9cb838f21611","Type":"ContainerStarted","Data":"cad4c44aa6f97f6add70c5201ab1a6795aa59a3c6351fc0ad97c100da7cf55a2"}
Oct 30 21:45:19.577858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.577677    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6" event={"ID":"a60f039f-582f-4178-916d-5bf0260ed803","Type":"ContainerStarted","Data":"67b49818bf6a3268f17b388afa10aed360984edb2dbe6cee6b109e07a91fb534"}
Oct 30 21:45:19.580813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.580785    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6" event={"ID":"83727964-4998-42ed-82b2-7ac611269201","Type":"ContainerStarted","Data":"703396618e012417723bd052514914d59728a6b6852469aedbad2b94ed8593e8"}
Oct 30 21:45:19.583593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.583568    4918 generic.go:356] "Generic (PLEG): container finished" podID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" containerID="b161fef17a17b3f6d762468832aa2be10ffb6559b6327816895adfa73f9c6d5a" exitCode=60
Oct 30 21:45:19.583698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.583650    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z" event={"ID":"79a3898f-6fd6-4db6-811b-48fdef8f81ba","Type":"ContainerDied","Data":"b161fef17a17b3f6d762468832aa2be10ffb6559b6327816895adfa73f9c6d5a"}
Oct 30 21:45:19.583759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.583708    4918 scope.go:122] "RemoveContainer" containerID="7fd3c1973498c040ec81d0e2f3f8230fa043192ce6f3e458b6d0321e379b6e40"
Oct 30 21:45:19.662575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.662081    4918 generic.go:356] "Generic (PLEG): container finished" podID="e65d8a21-107a-40ad-b8fc-967cb5008270" containerID="b9d52ca549490fb209f6dad63f269e9a042c855177af78fa3cf6b81a9cf94e73" exitCode=60
Oct 30 21:45:19.662575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.662315    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b" event={"ID":"e65d8a21-107a-40ad-b8fc-967cb5008270","Type":"ContainerDied","Data":"b9d52ca549490fb209f6dad63f269e9a042c855177af78fa3cf6b81a9cf94e73"}
Oct 30 21:45:19.667717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.667689    4918 generic.go:356] "Generic (PLEG): container finished" podID="c3712d27-9480-45ce-a089-caff86ddb3ec" containerID="a05c70cb41268942073d86e7d37dcc7d2e49234b60b0cfaed364ffc3857840dc" exitCode=60
Oct 30 21:45:19.668463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.667869    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg" event={"ID":"c3712d27-9480-45ce-a089-caff86ddb3ec","Type":"ContainerDied","Data":"a05c70cb41268942073d86e7d37dcc7d2e49234b60b0cfaed364ffc3857840dc"}
Oct 30 21:45:19.673444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.673369    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk" event={"ID":"ee00d03a-36a1-47fe-be99-0cf595abd3a5","Type":"ContainerStarted","Data":"78bea9c522b22986e178320266fe5eda05d25abfa4bdfd2266586da787a3085e"}
Oct 30 21:45:19.676725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.676702    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7" event={"ID":"6bba61f8-4422-4206-8723-40d9ac86a4c2","Type":"ContainerStarted","Data":"bae5ea0d58d4370ac1ba7ddb101945af28b3a669759b33f7487712e944cb30e8"}
Oct 30 21:45:19.755555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.755506    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"]
Oct 30 21:45:19.878138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:19.878113    4918 scope.go:122] "RemoveContainer" containerID="5a0558999cf6e2c2eead69207f38e48a9dad7d1abc3fec82969047ef3c3c1d61"
Oct 30 21:45:20.069959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.069925    4918 scope.go:122] "RemoveContainer" containerID="15a956bc024235f56ecbfdef23b6848a75252a7c1dc9738c290e14cbd94ed0a9"
Oct 30 21:45:20.071640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.071088    4918 scope.go:122] "RemoveContainer" containerID="0679ac3a391382a406b51965d332fc1d35113e00567b36fd783b42a279a378a1"
Oct 30 21:45:20.267601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.253937    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 30 21:45:20.268180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.254577    4918 scope.go:122] "RemoveContainer" containerID="e65069a6f0c0a70ff185401140b9f1ea9c6cc6d677d45041f5ab4c27671e0843"
Oct 30 21:45:20.268435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.268143    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 30 21:45:20.456300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.455896    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"
Oct 30 21:45:20.477180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.476712    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:20.478578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.478243    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:20.480334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.480316    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:45:20.558723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.558691    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"
Oct 30 21:45:20.563726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.563695    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"
Oct 30 21:45:20.630549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.628022    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3712d27-9480-45ce-a089-caff86ddb3ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3712d27-9480-45ce-a089-caff86ddb3ec-maas-api-service-ca\") pod \"c3712d27-9480-45ce-a089-caff86ddb3ec\" (UID: \"c3712d27-9480-45ce-a089-caff86ddb3ec\") "
Oct 30 21:45:20.630549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.628336    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79a3898f-6fd6-4db6-811b-48fdef8f81ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a3898f-6fd6-4db6-811b-48fdef8f81ba-maas-api-service-ca\") pod \"79a3898f-6fd6-4db6-811b-48fdef8f81ba\" (UID: \"79a3898f-6fd6-4db6-811b-48fdef8f81ba\") "
Oct 30 21:45:20.630549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.628467    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3712d27-9480-45ce-a089-caff86ddb3ec-kube-api-access-fs7tb\" (UniqueName: \"kubernetes.io/projected/c3712d27-9480-45ce-a089-caff86ddb3ec-kube-api-access-fs7tb\") pod \"c3712d27-9480-45ce-a089-caff86ddb3ec\" (UID: \"c3712d27-9480-45ce-a089-caff86ddb3ec\") "
Oct 30 21:45:20.630549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.628595    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65d8a21-107a-40ad-b8fc-967cb5008270-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65d8a21-107a-40ad-b8fc-967cb5008270-maas-api-service-ca\") pod \"e65d8a21-107a-40ad-b8fc-967cb5008270\" (UID: \"e65d8a21-107a-40ad-b8fc-967cb5008270\") "
Oct 30 21:45:20.630549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.628752    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3712d27-9480-45ce-a089-caff86ddb3ec-maas-api-service-ca" pod "c3712d27-9480-45ce-a089-caff86ddb3ec" (UID: "c3712d27-9480-45ce-a089-caff86ddb3ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:20.630549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.629094    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a3898f-6fd6-4db6-811b-48fdef8f81ba-maas-api-service-ca" pod "79a3898f-6fd6-4db6-811b-48fdef8f81ba" (UID: "79a3898f-6fd6-4db6-811b-48fdef8f81ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:20.630549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.629649    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a3898f-6fd6-4db6-811b-48fdef8f81ba-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:20.630549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.629672    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3712d27-9480-45ce-a089-caff86ddb3ec-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:20.630549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.629950    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65d8a21-107a-40ad-b8fc-967cb5008270-maas-api-service-ca" pod "e65d8a21-107a-40ad-b8fc-967cb5008270" (UID: "e65d8a21-107a-40ad-b8fc-967cb5008270"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:20.636450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.636425    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3712d27-9480-45ce-a089-caff86ddb3ec-kube-api-access-fs7tb" pod "c3712d27-9480-45ce-a089-caff86ddb3ec" (UID: "c3712d27-9480-45ce-a089-caff86ddb3ec"). InnerVolumeSpecName "kube-api-access-fs7tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:20.654444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.654410    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"]
Oct 30 21:45:20.730951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.730907    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65d8a21-107a-40ad-b8fc-967cb5008270-kube-api-access-b9fsd\" (UniqueName: \"kubernetes.io/projected/e65d8a21-107a-40ad-b8fc-967cb5008270-kube-api-access-b9fsd\") pod \"e65d8a21-107a-40ad-b8fc-967cb5008270\" (UID: \"e65d8a21-107a-40ad-b8fc-967cb5008270\") "
Oct 30 21:45:20.731229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.731209    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a3898f-6fd6-4db6-811b-48fdef8f81ba-kube-api-access-n279k\" (UniqueName: \"kubernetes.io/projected/79a3898f-6fd6-4db6-811b-48fdef8f81ba-kube-api-access-n279k\") pod \"79a3898f-6fd6-4db6-811b-48fdef8f81ba\" (UID: \"79a3898f-6fd6-4db6-811b-48fdef8f81ba\") "
Oct 30 21:45:20.732439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.732348    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs7tb\" (UniqueName: \"kubernetes.io/projected/c3712d27-9480-45ce-a089-caff86ddb3ec-kube-api-access-fs7tb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:20.732609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.732592    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65d8a21-107a-40ad-b8fc-967cb5008270-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:20.736300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.736272    4918 generic.go:356] "Generic (PLEG): container finished" podID="83727964-4998-42ed-82b2-7ac611269201" containerID="703396618e012417723bd052514914d59728a6b6852469aedbad2b94ed8593e8" exitCode=60
Oct 30 21:45:20.736611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.736589    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6" event={"ID":"83727964-4998-42ed-82b2-7ac611269201","Type":"ContainerDied","Data":"703396618e012417723bd052514914d59728a6b6852469aedbad2b94ed8593e8"}
Oct 30 21:45:20.736762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.736749    4918 scope.go:122] "RemoveContainer" containerID="8db3775c6566b63512f2f53c9e28e49f253da12ebd228d469960fd932ddf95c4"
Oct 30 21:45:20.737341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.737265    4918 scope.go:122] "RemoveContainer" containerID="703396618e012417723bd052514914d59728a6b6852469aedbad2b94ed8593e8"
Oct 30 21:45:20.737978 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:20.737851    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6_odh-ai-gateway-infra(83727964-4998-42ed-82b2-7ac611269201)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6" podUID="83727964-4998-42ed-82b2-7ac611269201"
Oct 30 21:45:20.738273 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.738247    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"]
Oct 30 21:45:20.742069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.741720    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a3898f-6fd6-4db6-811b-48fdef8f81ba-kube-api-access-n279k" pod "79a3898f-6fd6-4db6-811b-48fdef8f81ba" (UID: "79a3898f-6fd6-4db6-811b-48fdef8f81ba"). InnerVolumeSpecName "kube-api-access-n279k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:20.742969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.742945    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z" event={"ID":"79a3898f-6fd6-4db6-811b-48fdef8f81ba","Type":"ContainerDied","Data":"4393ed71decd2402d9a8de1cf29969f537798db669acae854605f85a1f88b689"}
Oct 30 21:45:20.743957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.743940    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"
Oct 30 21:45:20.747224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.746964    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b" event={"ID":"e65d8a21-107a-40ad-b8fc-967cb5008270","Type":"ContainerDied","Data":"cb34c88a58e0bb5e5dbd298efa206296dadd420201d30b47fd541d1e97341ef8"}
Oct 30 21:45:20.747224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.747057    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"
Oct 30 21:45:20.752565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.750994    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65d8a21-107a-40ad-b8fc-967cb5008270-kube-api-access-b9fsd" pod "e65d8a21-107a-40ad-b8fc-967cb5008270" (UID: "e65d8a21-107a-40ad-b8fc-967cb5008270"). InnerVolumeSpecName "kube-api-access-b9fsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:20.753782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.752989    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx" event={"ID":"5a319997-3bb4-419a-a27b-23849ee7691b","Type":"ContainerStarted","Data":"55828314bf1ab9b0a45f7f9bd20262d58fca3ab429abbb4f40cd3e8f3e5005ec"}
Oct 30 21:45:20.764523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.763812    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr" event={"ID":"57e008fe-d6b5-4328-a70d-188bdb928a4d","Type":"ContainerStarted","Data":"e5e04938ee2d2c124c63fd0e595d173ed83213925db91cd472e19cb492e672a1"}
Oct 30 21:45:20.766800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.766777    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"
Oct 30 21:45:20.766906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.766849    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg" event={"ID":"c3712d27-9480-45ce-a089-caff86ddb3ec","Type":"ContainerDied","Data":"408c46d7aaec7eff0aa8de6610fda04b171b9dafdfc580909a3c319bb7393599"}
Oct 30 21:45:20.767552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.767472    4918 scope.go:122] "RemoveContainer" containerID="b161fef17a17b3f6d762468832aa2be10ffb6559b6327816895adfa73f9c6d5a"
Oct 30 21:45:20.787075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.786961    4918 scope.go:122] "RemoveContainer" containerID="b9d52ca549490fb209f6dad63f269e9a042c855177af78fa3cf6b81a9cf94e73"
Oct 30 21:45:20.789601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.789574    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42" event={"ID":"409b4a57-a05d-49c5-92e0-58a4b1712afc","Type":"ContainerStarted","Data":"9e6b34b2ca819c47dc847740f6b627937099c058d245d4755bd5f1798dc81d22"}
Oct 30 21:45:20.793878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.793855    4918 generic.go:356] "Generic (PLEG): container finished" podID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" containerID="78bea9c522b22986e178320266fe5eda05d25abfa4bdfd2266586da787a3085e" exitCode=60
Oct 30 21:45:20.794073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.794055    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk" event={"ID":"ee00d03a-36a1-47fe-be99-0cf595abd3a5","Type":"ContainerDied","Data":"78bea9c522b22986e178320266fe5eda05d25abfa4bdfd2266586da787a3085e"}
Oct 30 21:45:20.794609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.794591    4918 scope.go:122] "RemoveContainer" containerID="78bea9c522b22986e178320266fe5eda05d25abfa4bdfd2266586da787a3085e"
Oct 30 21:45:20.795119 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:20.795096    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk_odh-ai-gateway-infra(ee00d03a-36a1-47fe-be99-0cf595abd3a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5"
Oct 30 21:45:20.802607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.800231    4918 generic.go:356] "Generic (PLEG): container finished" podID="6bba61f8-4422-4206-8723-40d9ac86a4c2" containerID="bae5ea0d58d4370ac1ba7ddb101945af28b3a669759b33f7487712e944cb30e8" exitCode=60
Oct 30 21:45:20.802607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.800405    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7" event={"ID":"6bba61f8-4422-4206-8723-40d9ac86a4c2","Type":"ContainerDied","Data":"bae5ea0d58d4370ac1ba7ddb101945af28b3a669759b33f7487712e944cb30e8"}
Oct 30 21:45:20.802607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.800856    4918 scope.go:122] "RemoveContainer" containerID="bae5ea0d58d4370ac1ba7ddb101945af28b3a669759b33f7487712e944cb30e8"
Oct 30 21:45:20.802607 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:20.801286    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7_odh-ai-gateway-infra(6bba61f8-4422-4206-8723-40d9ac86a4c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2"
Oct 30 21:45:20.839346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.839311    4918 generic.go:356] "Generic (PLEG): container finished" podID="aedc6368-2835-4821-bf6c-9cb838f21611" containerID="cad4c44aa6f97f6add70c5201ab1a6795aa59a3c6351fc0ad97c100da7cf55a2" exitCode=60
Oct 30 21:45:20.839651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.839628    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp" event={"ID":"aedc6368-2835-4821-bf6c-9cb838f21611","Type":"ContainerDied","Data":"cad4c44aa6f97f6add70c5201ab1a6795aa59a3c6351fc0ad97c100da7cf55a2"}
Oct 30 21:45:20.839771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.839683    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9fsd\" (UniqueName: \"kubernetes.io/projected/e65d8a21-107a-40ad-b8fc-967cb5008270-kube-api-access-b9fsd\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:20.839771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.839708    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n279k\" (UniqueName: \"kubernetes.io/projected/79a3898f-6fd6-4db6-811b-48fdef8f81ba-kube-api-access-n279k\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:20.843279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.843255    4918 generic.go:356] "Generic (PLEG): container finished" podID="a60f039f-582f-4178-916d-5bf0260ed803" containerID="67b49818bf6a3268f17b388afa10aed360984edb2dbe6cee6b109e07a91fb534" exitCode=60
Oct 30 21:45:20.844025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.843998    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6" event={"ID":"a60f039f-582f-4178-916d-5bf0260ed803","Type":"ContainerDied","Data":"67b49818bf6a3268f17b388afa10aed360984edb2dbe6cee6b109e07a91fb534"}
Oct 30 21:45:20.854472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.854449    4918 scope.go:122] "RemoveContainer" containerID="a05c70cb41268942073d86e7d37dcc7d2e49234b60b0cfaed364ffc3857840dc"
Oct 30 21:45:20.902438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.884533    4918 scope.go:122] "RemoveContainer" containerID="0fedd7fe84b435bc2fd23097e044ff9734520d2fcf23877b73c04de7d8d90096"
Oct 30 21:45:20.953512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:20.953482    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"]
Oct 30 21:45:21.128421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.101526    4918 scope.go:122] "RemoveContainer" containerID="8cfa7a50db8b7e393a04285f412301ccb3892dbc1beff4050d49c0481c0b062c"
Oct 30 21:45:21.264780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.224652    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 30 21:45:21.264780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.224709    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 30 21:45:21.279552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.279523    4918 scope.go:122] "RemoveContainer" containerID="b83481189ef09bda65f1973b330e1405df7fe27e45d4fdbd82279bd35904d936"
Oct 30 21:45:21.444722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.395971    4918 scope.go:122] "RemoveContainer" containerID="672ec0326efe71d27102dc1a3b7afaf244b7d257be9224eb0d29734937f976e8"
Oct 30 21:45:21.626265 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.624437    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"]
Oct 30 21:45:21.627970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.627944    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a378a2-29889945-rzr5z"]
Oct 30 21:45:21.683179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.682418    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"]
Oct 30 21:45:21.695637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.695136    4918 scope.go:122] "RemoveContainer" containerID="1300cc02b07c713f550535f2dde2ca35a083e9fb7302e7ba0e5420ff6000d611"
Oct 30 21:45:21.695637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.695249    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-36a099-29889945-njtmg"]
Oct 30 21:45:21.716530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.716505    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"
Oct 30 21:45:21.727573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.727548    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"]
Oct 30 21:45:21.734493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.734467    4918 scope.go:122] "RemoveContainer" containerID="d7943d39b1c8c61643c668cb64ceaebbfa2894011129d254b6daa4a21a2353bd"
Oct 30 21:45:21.745435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.742347    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60f039f-582f-4178-916d-5bf0260ed803-kube-api-access-xf9d9\" (UniqueName: \"kubernetes.io/projected/a60f039f-582f-4178-916d-5bf0260ed803-kube-api-access-xf9d9\") pod \"a60f039f-582f-4178-916d-5bf0260ed803\" (UID: \"a60f039f-582f-4178-916d-5bf0260ed803\") "
Oct 30 21:45:21.758300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.758221    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60f039f-582f-4178-916d-5bf0260ed803-kube-api-access-xf9d9" pod "a60f039f-582f-4178-916d-5bf0260ed803" (UID: "a60f039f-582f-4178-916d-5bf0260ed803"). InnerVolumeSpecName "kube-api-access-xf9d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:21.761554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.761525    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60f039f-582f-4178-916d-5bf0260ed803-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60f039f-582f-4178-916d-5bf0260ed803-maas-api-service-ca\") pod \"a60f039f-582f-4178-916d-5bf0260ed803\" (UID: \"a60f039f-582f-4178-916d-5bf0260ed803\") "
Oct 30 21:45:21.764336 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.762753    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60f039f-582f-4178-916d-5bf0260ed803-maas-api-service-ca" pod "a60f039f-582f-4178-916d-5bf0260ed803" (UID: "a60f039f-582f-4178-916d-5bf0260ed803"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:21.766663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.766521    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60f039f-582f-4178-916d-5bf0260ed803-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:21.766663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.766553    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9d9\" (UniqueName: \"kubernetes.io/projected/a60f039f-582f-4178-916d-5bf0260ed803-kube-api-access-xf9d9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:21.804145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.803975    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"]
Oct 30 21:45:21.813931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.813894    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"
Oct 30 21:45:21.828658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.828629    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"]
Oct 30 21:45:21.833744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.833718    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-384164-29889945-lhh4b"]
Oct 30 21:45:21.842441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.842422    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"]
Oct 30 21:45:21.856557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.856529    4918 generic.go:356] "Generic (PLEG): container finished" podID="f6b82123-e7fd-41ff-9655-c2610e1973bf" containerID="0386e5b2fb187628a0169e84e04fdba8450e52ef82e542130ed175193ea63ebe" exitCode=60
Oct 30 21:45:21.856689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.856616    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j" event={"ID":"f6b82123-e7fd-41ff-9655-c2610e1973bf","Type":"ContainerDied","Data":"0386e5b2fb187628a0169e84e04fdba8450e52ef82e542130ed175193ea63ebe"}
Oct 30 21:45:21.856753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.856693    4918 scope.go:122] "RemoveContainer" containerID="15a956bc024235f56ecbfdef23b6848a75252a7c1dc9738c290e14cbd94ed0a9"
Oct 30 21:45:21.857238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.857219    4918 scope.go:122] "RemoveContainer" containerID="0386e5b2fb187628a0169e84e04fdba8450e52ef82e542130ed175193ea63ebe"
Oct 30 21:45:21.857883 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:21.857853    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j_odh-ai-gateway-infra(f6b82123-e7fd-41ff-9655-c2610e1973bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j" podUID="f6b82123-e7fd-41ff-9655-c2610e1973bf"
Oct 30 21:45:21.860117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.860094    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w" event={"ID":"06cfd24a-7880-40a4-98ed-0c9326491795","Type":"ContainerStarted","Data":"0af9ed300413693698053791469002e6c63e4f0a0fa12c342bdddc785097c1db"}
Oct 30 21:45:21.874332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.872480    4918 generic.go:356] "Generic (PLEG): container finished" podID="57e008fe-d6b5-4328-a70d-188bdb928a4d" containerID="e5e04938ee2d2c124c63fd0e595d173ed83213925db91cd472e19cb492e672a1" exitCode=60
Oct 30 21:45:21.874332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.872630    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr" event={"ID":"57e008fe-d6b5-4328-a70d-188bdb928a4d","Type":"ContainerDied","Data":"e5e04938ee2d2c124c63fd0e595d173ed83213925db91cd472e19cb492e672a1"}
Oct 30 21:45:21.874332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.873375    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aedc6368-2835-4821-bf6c-9cb838f21611-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedc6368-2835-4821-bf6c-9cb838f21611-maas-api-service-ca\") pod \"aedc6368-2835-4821-bf6c-9cb838f21611\" (UID: \"aedc6368-2835-4821-bf6c-9cb838f21611\") "
Oct 30 21:45:21.874332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.873572    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aedc6368-2835-4821-bf6c-9cb838f21611-kube-api-access-scd68\" (UniqueName: \"kubernetes.io/projected/aedc6368-2835-4821-bf6c-9cb838f21611-kube-api-access-scd68\") pod \"aedc6368-2835-4821-bf6c-9cb838f21611\" (UID: \"aedc6368-2835-4821-bf6c-9cb838f21611\") "
Oct 30 21:45:21.874332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.874128    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aedc6368-2835-4821-bf6c-9cb838f21611-maas-api-service-ca" pod "aedc6368-2835-4821-bf6c-9cb838f21611" (UID: "aedc6368-2835-4821-bf6c-9cb838f21611"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:21.874826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.874798    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedc6368-2835-4821-bf6c-9cb838f21611-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:21.878583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.878559    4918 scope.go:122] "RemoveContainer" containerID="e4e09214c9a7354ae230fee9ce6d438787a3d451188d1a388e24135ac8685940"
Oct 30 21:45:21.881725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.881698    4918 generic.go:356] "Generic (PLEG): container finished" podID="409b4a57-a05d-49c5-92e0-58a4b1712afc" containerID="9e6b34b2ca819c47dc847740f6b627937099c058d245d4755bd5f1798dc81d22" exitCode=60
Oct 30 21:45:21.882458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.881784    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42" event={"ID":"409b4a57-a05d-49c5-92e0-58a4b1712afc","Type":"ContainerDied","Data":"9e6b34b2ca819c47dc847740f6b627937099c058d245d4755bd5f1798dc81d22"}
Oct 30 21:45:21.882914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.882892    4918 scope.go:122] "RemoveContainer" containerID="9e6b34b2ca819c47dc847740f6b627937099c058d245d4755bd5f1798dc81d22"
Oct 30 21:45:21.883325 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:21.883299    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42_odh-ai-gateway-infra(409b4a57-a05d-49c5-92e0-58a4b1712afc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc"
Oct 30 21:45:21.885200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.885179    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"]
Oct 30 21:45:21.891630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.891606    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedc6368-2835-4821-bf6c-9cb838f21611-kube-api-access-scd68" pod "aedc6368-2835-4821-bf6c-9cb838f21611" (UID: "aedc6368-2835-4821-bf6c-9cb838f21611"). InnerVolumeSpecName "kube-api-access-scd68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:21.900014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.897334    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp" event={"ID":"aedc6368-2835-4821-bf6c-9cb838f21611","Type":"ContainerDied","Data":"0ab7724b37708f89abd37e6efc68e80ce94ba6d728319a54c2b567fa518b74ce"}
Oct 30 21:45:21.900014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.897484    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"
Oct 30 21:45:21.918140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.918114    4918 scope.go:122] "RemoveContainer" containerID="375c23e52f2a61533475d58acba8368845d69f773a1c49b21d63cdcaff99db2e"
Oct 30 21:45:21.921466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.920458    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6" event={"ID":"a60f039f-582f-4178-916d-5bf0260ed803","Type":"ContainerDied","Data":"5db8c16347b3b4ec2bdeb561fe425ccefb1319b058d46f73943cf42fb05914c7"}
Oct 30 21:45:21.921466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.920585    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"
Oct 30 21:45:21.979992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.977865    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scd68\" (UniqueName: \"kubernetes.io/projected/aedc6368-2835-4821-bf6c-9cb838f21611-kube-api-access-scd68\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:21.982491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.981415    4918 generic.go:356] "Generic (PLEG): container finished" podID="5a319997-3bb4-419a-a27b-23849ee7691b" containerID="55828314bf1ab9b0a45f7f9bd20262d58fca3ab429abbb4f40cd3e8f3e5005ec" exitCode=60
Oct 30 21:45:21.983724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.982846    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx" event={"ID":"5a319997-3bb4-419a-a27b-23849ee7691b","Type":"ContainerDied","Data":"55828314bf1ab9b0a45f7f9bd20262d58fca3ab429abbb4f40cd3e8f3e5005ec"}
Oct 30 21:45:21.983724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:21.983616    4918 scope.go:122] "RemoveContainer" containerID="cad4c44aa6f97f6add70c5201ab1a6795aa59a3c6351fc0ad97c100da7cf55a2"
Oct 30 21:45:22.067507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.037204    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"]
Oct 30 21:45:22.067507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.054129    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a9230f-29889945-7zdmp"]
Oct 30 21:45:22.070849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.069785    4918 scope.go:122] "RemoveContainer" containerID="66db99a02ad0afbfc774e06d30c398595426ab1cf161c7c2bc7f64b0e0a37fa0"
Oct 30 21:45:22.071024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.070867    4918 scope.go:122] "RemoveContainer" containerID="a3f8d14c2b2ae504ca50213928c1c6ad7bd18c2d0eaa7fca4de065267daed6ae"
Oct 30 21:45:22.084586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.084565    4918 scope.go:122] "RemoveContainer" containerID="67b49818bf6a3268f17b388afa10aed360984edb2dbe6cee6b109e07a91fb534"
Oct 30 21:45:22.118454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.118434    4918 scope.go:122] "RemoveContainer" containerID="6dd25e3e03dce0854383c9fe95c1d35112a07e2541811710850b4c6bdfd84232"
Oct 30 21:45:22.148838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.148813    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"]
Oct 30 21:45:22.164455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.164428    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-7b5a7c-29889945-rmkb6"]
Oct 30 21:45:22.231964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.231933    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:22.233454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.233430    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:22.237184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.236870    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:45:22.241336 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.241316    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"
Oct 30 21:45:22.286037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.285951    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee00d03a-36a1-47fe-be99-0cf595abd3a5-kube-api-access-9rkd4\" (UniqueName: \"kubernetes.io/projected/ee00d03a-36a1-47fe-be99-0cf595abd3a5-kube-api-access-9rkd4\") pod \"ee00d03a-36a1-47fe-be99-0cf595abd3a5\" (UID: \"ee00d03a-36a1-47fe-be99-0cf595abd3a5\") "
Oct 30 21:45:22.286793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.286768    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee00d03a-36a1-47fe-be99-0cf595abd3a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee00d03a-36a1-47fe-be99-0cf595abd3a5-maas-api-service-ca\") pod \"ee00d03a-36a1-47fe-be99-0cf595abd3a5\" (UID: \"ee00d03a-36a1-47fe-be99-0cf595abd3a5\") "
Oct 30 21:45:22.292627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.292036    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee00d03a-36a1-47fe-be99-0cf595abd3a5-kube-api-access-9rkd4" pod "ee00d03a-36a1-47fe-be99-0cf595abd3a5" (UID: "ee00d03a-36a1-47fe-be99-0cf595abd3a5"). InnerVolumeSpecName "kube-api-access-9rkd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:22.294894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.293058    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee00d03a-36a1-47fe-be99-0cf595abd3a5-maas-api-service-ca" pod "ee00d03a-36a1-47fe-be99-0cf595abd3a5" (UID: "ee00d03a-36a1-47fe-be99-0cf595abd3a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:22.378741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.378573    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"]
Oct 30 21:45:22.390324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.390260    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rkd4\" (UniqueName: \"kubernetes.io/projected/ee00d03a-36a1-47fe-be99-0cf595abd3a5-kube-api-access-9rkd4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:22.390324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.390297    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee00d03a-36a1-47fe-be99-0cf595abd3a5-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:22.550571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.548008    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"
Oct 30 21:45:22.560501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.560473    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"
Oct 30 21:45:22.570491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.570457    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"
Oct 30 21:45:22.573192 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.573176    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"
Oct 30 21:45:22.695034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.695003    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83727964-4998-42ed-82b2-7ac611269201-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83727964-4998-42ed-82b2-7ac611269201-maas-api-service-ca\") pod \"83727964-4998-42ed-82b2-7ac611269201\" (UID: \"83727964-4998-42ed-82b2-7ac611269201\") "
Oct 30 21:45:22.695205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.695086    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a319997-3bb4-419a-a27b-23849ee7691b-kube-api-access-hz8wx\" (UniqueName: \"kubernetes.io/projected/5a319997-3bb4-419a-a27b-23849ee7691b-kube-api-access-hz8wx\") pod \"5a319997-3bb4-419a-a27b-23849ee7691b\" (UID: \"5a319997-3bb4-419a-a27b-23849ee7691b\") "
Oct 30 21:45:22.695205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.695184    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a319997-3bb4-419a-a27b-23849ee7691b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a319997-3bb4-419a-a27b-23849ee7691b-maas-api-service-ca\") pod \"5a319997-3bb4-419a-a27b-23849ee7691b\" (UID: \"5a319997-3bb4-419a-a27b-23849ee7691b\") "
Oct 30 21:45:22.695361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.695235    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bba61f8-4422-4206-8723-40d9ac86a4c2-kube-api-access-r9stg\" (UniqueName: \"kubernetes.io/projected/6bba61f8-4422-4206-8723-40d9ac86a4c2-kube-api-access-r9stg\") pod \"6bba61f8-4422-4206-8723-40d9ac86a4c2\" (UID: \"6bba61f8-4422-4206-8723-40d9ac86a4c2\") "
Oct 30 21:45:22.695361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.695305    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e008fe-d6b5-4328-a70d-188bdb928a4d-kube-api-access-p8qcp\" (UniqueName: \"kubernetes.io/projected/57e008fe-d6b5-4328-a70d-188bdb928a4d-kube-api-access-p8qcp\") pod \"57e008fe-d6b5-4328-a70d-188bdb928a4d\" (UID: \"57e008fe-d6b5-4328-a70d-188bdb928a4d\") "
Oct 30 21:45:22.695502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.695352    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83727964-4998-42ed-82b2-7ac611269201-maas-api-service-ca" pod "83727964-4998-42ed-82b2-7ac611269201" (UID: "83727964-4998-42ed-82b2-7ac611269201"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:22.695675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.695652    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57e008fe-d6b5-4328-a70d-188bdb928a4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e008fe-d6b5-4328-a70d-188bdb928a4d-maas-api-service-ca\") pod \"57e008fe-d6b5-4328-a70d-188bdb928a4d\" (UID: \"57e008fe-d6b5-4328-a70d-188bdb928a4d\") "
Oct 30 21:45:22.695892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.695872    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83727964-4998-42ed-82b2-7ac611269201-kube-api-access-pbtf4\" (UniqueName: \"kubernetes.io/projected/83727964-4998-42ed-82b2-7ac611269201-kube-api-access-pbtf4\") pod \"83727964-4998-42ed-82b2-7ac611269201\" (UID: \"83727964-4998-42ed-82b2-7ac611269201\") "
Oct 30 21:45:22.696350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.696020    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bba61f8-4422-4206-8723-40d9ac86a4c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bba61f8-4422-4206-8723-40d9ac86a4c2-maas-api-service-ca\") pod \"6bba61f8-4422-4206-8723-40d9ac86a4c2\" (UID: \"6bba61f8-4422-4206-8723-40d9ac86a4c2\") "
Oct 30 21:45:22.696350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.695661    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a319997-3bb4-419a-a27b-23849ee7691b-maas-api-service-ca" pod "5a319997-3bb4-419a-a27b-23849ee7691b" (UID: "5a319997-3bb4-419a-a27b-23849ee7691b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:22.696350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.695959    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e008fe-d6b5-4328-a70d-188bdb928a4d-maas-api-service-ca" pod "57e008fe-d6b5-4328-a70d-188bdb928a4d" (UID: "57e008fe-d6b5-4328-a70d-188bdb928a4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:22.696350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.696331    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bba61f8-4422-4206-8723-40d9ac86a4c2-maas-api-service-ca" pod "6bba61f8-4422-4206-8723-40d9ac86a4c2" (UID: "6bba61f8-4422-4206-8723-40d9ac86a4c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:22.697109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.697029    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83727964-4998-42ed-82b2-7ac611269201-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:22.697109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.697064    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a319997-3bb4-419a-a27b-23849ee7691b-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:22.697109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.697080    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e008fe-d6b5-4328-a70d-188bdb928a4d-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:22.697109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.697095    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bba61f8-4422-4206-8723-40d9ac86a4c2-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:22.786828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.786793    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a319997-3bb4-419a-a27b-23849ee7691b-kube-api-access-hz8wx" pod "5a319997-3bb4-419a-a27b-23849ee7691b" (UID: "5a319997-3bb4-419a-a27b-23849ee7691b"). InnerVolumeSpecName "kube-api-access-hz8wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:22.786998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.786874    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e008fe-d6b5-4328-a70d-188bdb928a4d-kube-api-access-p8qcp" pod "57e008fe-d6b5-4328-a70d-188bdb928a4d" (UID: "57e008fe-d6b5-4328-a70d-188bdb928a4d"). InnerVolumeSpecName "kube-api-access-p8qcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:22.786998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.786918    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83727964-4998-42ed-82b2-7ac611269201-kube-api-access-pbtf4" pod "83727964-4998-42ed-82b2-7ac611269201" (UID: "83727964-4998-42ed-82b2-7ac611269201"). InnerVolumeSpecName "kube-api-access-pbtf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:22.787227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.787211    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bba61f8-4422-4206-8723-40d9ac86a4c2-kube-api-access-r9stg" pod "6bba61f8-4422-4206-8723-40d9ac86a4c2" (UID: "6bba61f8-4422-4206-8723-40d9ac86a4c2"). InnerVolumeSpecName "kube-api-access-r9stg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:22.799092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.799071    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbtf4\" (UniqueName: \"kubernetes.io/projected/83727964-4998-42ed-82b2-7ac611269201-kube-api-access-pbtf4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:22.799181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.799099    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz8wx\" (UniqueName: \"kubernetes.io/projected/5a319997-3bb4-419a-a27b-23849ee7691b-kube-api-access-hz8wx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:22.799181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.799114    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9stg\" (UniqueName: \"kubernetes.io/projected/6bba61f8-4422-4206-8723-40d9ac86a4c2-kube-api-access-r9stg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:22.799181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.799122    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8qcp\" (UniqueName: \"kubernetes.io/projected/57e008fe-d6b5-4328-a70d-188bdb928a4d-kube-api-access-p8qcp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:22.942057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.942028    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"]
Oct 30 21:45:22.978201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.978169    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"]
Oct 30 21:45:22.997882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.997857    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"
Oct 30 21:45:22.998499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.998472    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6" event={"ID":"83727964-4998-42ed-82b2-7ac611269201","Type":"ContainerDied","Data":"8fdfd944f4100ef272fa63aaa1f45fd04a4afc93d9db74f57a71ecc358828c5f"}
Oct 30 21:45:22.998663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:22.998646    4918 scope.go:122] "RemoveContainer" containerID="703396618e012417723bd052514914d59728a6b6852469aedbad2b94ed8593e8"
Oct 30 21:45:23.000142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.000121    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"
Oct 30 21:45:23.000225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.000133    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx" event={"ID":"5a319997-3bb4-419a-a27b-23849ee7691b","Type":"ContainerDied","Data":"c3f32d113f50428bf89cd5c63974aa483ef6d0ad55754fc774fe67f387b1efa1"}
Oct 30 21:45:23.006556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.006451    4918 generic.go:356] "Generic (PLEG): container finished" podID="06cfd24a-7880-40a4-98ed-0c9326491795" containerID="0af9ed300413693698053791469002e6c63e4f0a0fa12c342bdddc785097c1db" exitCode=60
Oct 30 21:45:23.006556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.006483    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w" event={"ID":"06cfd24a-7880-40a4-98ed-0c9326491795","Type":"ContainerDied","Data":"0af9ed300413693698053791469002e6c63e4f0a0fa12c342bdddc785097c1db"}
Oct 30 21:45:23.008282 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.008260    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"
Oct 30 21:45:23.008379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.008258    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr" event={"ID":"57e008fe-d6b5-4328-a70d-188bdb928a4d","Type":"ContainerDied","Data":"621e5c5a02cde2c04965cbc545759428fe27aacb26116e26479b435399003a27"}
Oct 30 21:45:23.010654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.010631    4918 generic.go:356] "Generic (PLEG): container finished" podID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" containerID="224685a1c9a7c7dbc103e95cc83d8c1288a2db692862e9bb170f32ef468ca801" exitCode=60
Oct 30 21:45:23.010744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.010698    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs" event={"ID":"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af","Type":"ContainerDied","Data":"224685a1c9a7c7dbc103e95cc83d8c1288a2db692862e9bb170f32ef468ca801"}
Oct 30 21:45:23.011131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.011111    4918 scope.go:122] "RemoveContainer" containerID="224685a1c9a7c7dbc103e95cc83d8c1288a2db692862e9bb170f32ef468ca801"
Oct 30 21:45:23.011504 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:23.011481    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs_odh-ai-gateway-infra(e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af"
Oct 30 21:45:23.012847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.012825    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd" event={"ID":"2c42b67a-e483-4511-9be1-005d53f02bdc","Type":"ContainerStarted","Data":"30e8f8884e5df451ac3741c23b686d9865948bed9c05e9e877b796336c057467"}
Oct 30 21:45:23.014881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.014861    4918 generic.go:356] "Generic (PLEG): container finished" podID="4b80735b-3681-4774-b4a3-821f37748327" containerID="9d0b3bf026e0379b3d3567f42ea2386f9db42b7bbba11e39a0c2360c1d5c26f2" exitCode=60
Oct 30 21:45:23.014982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.014932    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj" event={"ID":"4b80735b-3681-4774-b4a3-821f37748327","Type":"ContainerDied","Data":"9d0b3bf026e0379b3d3567f42ea2386f9db42b7bbba11e39a0c2360c1d5c26f2"}
Oct 30 21:45:23.015361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.015315    4918 scope.go:122] "RemoveContainer" containerID="9d0b3bf026e0379b3d3567f42ea2386f9db42b7bbba11e39a0c2360c1d5c26f2"
Oct 30 21:45:23.015722 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:23.015670    4918 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj_odh-ai-gateway-infra(4b80735b-3681-4774-b4a3-821f37748327)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj" podUID="4b80735b-3681-4774-b4a3-821f37748327"
Oct 30 21:45:23.017346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.017322    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6" event={"ID":"296ad3ea-d106-480c-b05c-2c44452d39ca","Type":"ContainerStarted","Data":"d1a40ec8ec1227318cb78bc091b22167bb5b9127213a803127799cac24ddc24d"}
Oct 30 21:45:23.021247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.021223    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"
Oct 30 21:45:23.021344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.021231    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk" event={"ID":"ee00d03a-36a1-47fe-be99-0cf595abd3a5","Type":"ContainerDied","Data":"b9868bc52c48d75df907ff9ee155caa73c24f562593b79444b4cd05decfdfbc2"}
Oct 30 21:45:23.024449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.023731    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7" event={"ID":"6bba61f8-4422-4206-8723-40d9ac86a4c2","Type":"ContainerDied","Data":"548473a7f766c119ff7087ac3be46aed7faf103373c2e5b01f349b660b321468"}
Oct 30 21:45:23.024449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.023826    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"
Oct 30 21:45:23.054771 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.054703    4918 scope.go:122] "RemoveContainer" containerID="55828314bf1ab9b0a45f7f9bd20262d58fca3ab429abbb4f40cd3e8f3e5005ec"
Oct 30 21:45:23.085574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.084664    4918 scope.go:122] "RemoveContainer" containerID="0679ac3a391382a406b51965d332fc1d35113e00567b36fd783b42a279a378a1"
Oct 30 21:45:23.096269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.096238    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" path="/var/lib/kubelet/pods/79a3898f-6fd6-4db6-811b-48fdef8f81ba/volumes"
Oct 30 21:45:23.097136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.096994    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60f039f-582f-4178-916d-5bf0260ed803" path="/var/lib/kubelet/pods/a60f039f-582f-4178-916d-5bf0260ed803/volumes"
Oct 30 21:45:23.097669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.097645    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aedc6368-2835-4821-bf6c-9cb838f21611" path="/var/lib/kubelet/pods/aedc6368-2835-4821-bf6c-9cb838f21611/volumes"
Oct 30 21:45:23.098576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.098558    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3712d27-9480-45ce-a089-caff86ddb3ec" path="/var/lib/kubelet/pods/c3712d27-9480-45ce-a089-caff86ddb3ec/volumes"
Oct 30 21:45:23.099005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.098988    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65d8a21-107a-40ad-b8fc-967cb5008270" path="/var/lib/kubelet/pods/e65d8a21-107a-40ad-b8fc-967cb5008270/volumes"
Oct 30 21:45:23.239843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.239815    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"]
Oct 30 21:45:23.249491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.249462    4918 scope.go:122] "RemoveContainer" containerID="e5e04938ee2d2c124c63fd0e595d173ed83213925db91cd472e19cb492e672a1"
Oct 30 21:45:23.255850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.255823    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"]
Oct 30 21:45:23.265748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.265730    4918 scope.go:122] "RemoveContainer" containerID="8cfa7a50db8b7e393a04285f412301ccb3892dbc1beff4050d49c0481c0b062c"
Oct 30 21:45:23.319019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.318989    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"]
Oct 30 21:45:23.326498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.326471    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-729802-29889945-xpkdk"]
Oct 30 21:45:23.346139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.346086    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"]
Oct 30 21:45:23.355543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.355301    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15e79c-29889945-g5gc7"]
Oct 30 21:45:23.393108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.393072    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"]
Oct 30 21:45:23.400307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.400279    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-8072a0-29889945-nwzw6"]
Oct 30 21:45:23.427119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.427090    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"]
Oct 30 21:45:23.443406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.443365    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-726a18-29889945-8z2tr"]
Oct 30 21:45:23.460826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.460804    4918 scope.go:122] "RemoveContainer" containerID="b83481189ef09bda65f1973b330e1405df7fe27e45d4fdbd82279bd35904d936"
Oct 30 21:45:23.462537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.461839    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"]
Oct 30 21:45:23.467218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.467199    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"
Oct 30 21:45:23.471901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.471878    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"
Oct 30 21:45:23.475770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.475748    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-e596df-29889945-52lgx"]
Oct 30 21:45:23.475853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.475785    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"
Oct 30 21:45:23.479692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.479676    4918 scope.go:122] "RemoveContainer" containerID="78bea9c522b22986e178320266fe5eda05d25abfa4bdfd2266586da787a3085e"
Oct 30 21:45:23.492597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.492573    4918 scope.go:122] "RemoveContainer" containerID="bae5ea0d58d4370ac1ba7ddb101945af28b3a669759b33f7487712e944cb30e8"
Oct 30 21:45:23.524099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.524070    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409b4a57-a05d-49c5-92e0-58a4b1712afc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/409b4a57-a05d-49c5-92e0-58a4b1712afc-maas-api-service-ca\") pod \"409b4a57-a05d-49c5-92e0-58a4b1712afc\" (UID: \"409b4a57-a05d-49c5-92e0-58a4b1712afc\") "
Oct 30 21:45:23.524269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.524197    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409b4a57-a05d-49c5-92e0-58a4b1712afc-kube-api-access-r59jv\" (UniqueName: \"kubernetes.io/projected/409b4a57-a05d-49c5-92e0-58a4b1712afc-kube-api-access-r59jv\") pod \"409b4a57-a05d-49c5-92e0-58a4b1712afc\" (UID: \"409b4a57-a05d-49c5-92e0-58a4b1712afc\") "
Oct 30 21:45:23.524352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.524331    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06cfd24a-7880-40a4-98ed-0c9326491795-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cfd24a-7880-40a4-98ed-0c9326491795-maas-api-service-ca\") pod \"06cfd24a-7880-40a4-98ed-0c9326491795\" (UID: \"06cfd24a-7880-40a4-98ed-0c9326491795\") "
Oct 30 21:45:23.524627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.524607    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b82123-e7fd-41ff-9655-c2610e1973bf-kube-api-access-9qv2g\" (UniqueName: \"kubernetes.io/projected/f6b82123-e7fd-41ff-9655-c2610e1973bf-kube-api-access-9qv2g\") pod \"f6b82123-e7fd-41ff-9655-c2610e1973bf\" (UID: \"f6b82123-e7fd-41ff-9655-c2610e1973bf\") "
Oct 30 21:45:23.524822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.524802    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b82123-e7fd-41ff-9655-c2610e1973bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b82123-e7fd-41ff-9655-c2610e1973bf-maas-api-service-ca\") pod \"f6b82123-e7fd-41ff-9655-c2610e1973bf\" (UID: \"f6b82123-e7fd-41ff-9655-c2610e1973bf\") "
Oct 30 21:45:23.524822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.524811    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409b4a57-a05d-49c5-92e0-58a4b1712afc-maas-api-service-ca" pod "409b4a57-a05d-49c5-92e0-58a4b1712afc" (UID: "409b4a57-a05d-49c5-92e0-58a4b1712afc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:23.524955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.524935    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06cfd24a-7880-40a4-98ed-0c9326491795-kube-api-access-xkxxx\" (UniqueName: \"kubernetes.io/projected/06cfd24a-7880-40a4-98ed-0c9326491795-kube-api-access-xkxxx\") pod \"06cfd24a-7880-40a4-98ed-0c9326491795\" (UID: \"06cfd24a-7880-40a4-98ed-0c9326491795\") "
Oct 30 21:45:23.525491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.525453    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06cfd24a-7880-40a4-98ed-0c9326491795-maas-api-service-ca" pod "06cfd24a-7880-40a4-98ed-0c9326491795" (UID: "06cfd24a-7880-40a4-98ed-0c9326491795"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:23.525588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.525513    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b82123-e7fd-41ff-9655-c2610e1973bf-maas-api-service-ca" pod "f6b82123-e7fd-41ff-9655-c2610e1973bf" (UID: "f6b82123-e7fd-41ff-9655-c2610e1973bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:23.527653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.527631    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/409b4a57-a05d-49c5-92e0-58a4b1712afc-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:23.527745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.527656    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06cfd24a-7880-40a4-98ed-0c9326491795-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:23.527745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.527674    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b82123-e7fd-41ff-9655-c2610e1973bf-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:23.528312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.528265    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409b4a57-a05d-49c5-92e0-58a4b1712afc-kube-api-access-r59jv" pod "409b4a57-a05d-49c5-92e0-58a4b1712afc" (UID: "409b4a57-a05d-49c5-92e0-58a4b1712afc"). InnerVolumeSpecName "kube-api-access-r59jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:23.528550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.528528    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b82123-e7fd-41ff-9655-c2610e1973bf-kube-api-access-9qv2g" pod "f6b82123-e7fd-41ff-9655-c2610e1973bf" (UID: "f6b82123-e7fd-41ff-9655-c2610e1973bf"). InnerVolumeSpecName "kube-api-access-9qv2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:23.530353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.530318    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06cfd24a-7880-40a4-98ed-0c9326491795-kube-api-access-xkxxx" pod "06cfd24a-7880-40a4-98ed-0c9326491795" (UID: "06cfd24a-7880-40a4-98ed-0c9326491795"). InnerVolumeSpecName "kube-api-access-xkxxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:23.647167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.647131    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qv2g\" (UniqueName: \"kubernetes.io/projected/f6b82123-e7fd-41ff-9655-c2610e1973bf-kube-api-access-9qv2g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:23.647424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.647407    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkxxx\" (UniqueName: \"kubernetes.io/projected/06cfd24a-7880-40a4-98ed-0c9326491795-kube-api-access-xkxxx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:23.647528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:23.647516    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r59jv\" (UniqueName: \"kubernetes.io/projected/409b4a57-a05d-49c5-92e0-58a4b1712afc-kube-api-access-r59jv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:24.054254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.054218    4918 generic.go:356] "Generic (PLEG): container finished" podID="2c42b67a-e483-4511-9be1-005d53f02bdc" containerID="30e8f8884e5df451ac3741c23b686d9865948bed9c05e9e877b796336c057467" exitCode=60
Oct 30 21:45:24.054254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.054242    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd" event={"ID":"2c42b67a-e483-4511-9be1-005d53f02bdc","Type":"ContainerDied","Data":"30e8f8884e5df451ac3741c23b686d9865948bed9c05e9e877b796336c057467"}
Oct 30 21:45:24.054540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.054310    4918 scope.go:122] "RemoveContainer" containerID="66db99a02ad0afbfc774e06d30c398595426ab1cf161c7c2bc7f64b0e0a37fa0"
Oct 30 21:45:24.058806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.058781    4918 generic.go:356] "Generic (PLEG): container finished" podID="296ad3ea-d106-480c-b05c-2c44452d39ca" containerID="d1a40ec8ec1227318cb78bc091b22167bb5b9127213a803127799cac24ddc24d" exitCode=60
Oct 30 21:45:24.058948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.058865    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6" event={"ID":"296ad3ea-d106-480c-b05c-2c44452d39ca","Type":"ContainerDied","Data":"d1a40ec8ec1227318cb78bc091b22167bb5b9127213a803127799cac24ddc24d"}
Oct 30 21:45:24.060758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.060730    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"
Oct 30 21:45:24.060885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.060764    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42" event={"ID":"409b4a57-a05d-49c5-92e0-58a4b1712afc","Type":"ContainerDied","Data":"975bed7f259632bcc636c518fb24397eaaa79cc1d69a999b60cf599e91512064"}
Oct 30 21:45:24.111724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.111697    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"
Oct 30 21:45:24.112501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.111712    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j" event={"ID":"f6b82123-e7fd-41ff-9655-c2610e1973bf","Type":"ContainerDied","Data":"4c6619e1240af99aa4490920a800f0bc4ccf13f0f4cfe2684ed51dc5c7560955"}
Oct 30 21:45:24.112652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.112633    4918 scope.go:122] "RemoveContainer" containerID="a3f8d14c2b2ae504ca50213928c1c6ad7bd18c2d0eaa7fca4de065267daed6ae"
Oct 30 21:45:24.120574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.115957    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w" event={"ID":"06cfd24a-7880-40a4-98ed-0c9326491795","Type":"ContainerDied","Data":"a3e83cbc7495b5d9fc4a37675e71de356cf8a38b94908549118790aad765563f"}
Oct 30 21:45:24.120574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.116077    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"
Oct 30 21:45:24.145500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.145471    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"]
Oct 30 21:45:24.149573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.149552    4918 scope.go:122] "RemoveContainer" containerID="9e6b34b2ca819c47dc847740f6b627937099c058d245d4755bd5f1798dc81d22"
Oct 30 21:45:24.152789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.151628    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-fb2b58-29889945-sgf42"]
Oct 30 21:45:24.191412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.175257    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"]
Oct 30 21:45:24.200621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.200596    4918 scope.go:122] "RemoveContainer" containerID="0386e5b2fb187628a0169e84e04fdba8450e52ef82e542130ed175193ea63ebe"
Oct 30 21:45:24.212513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.212483    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bfc50e-29889945-ksh8w"]
Oct 30 21:45:24.218272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.218247    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"]
Oct 30 21:45:24.223116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.222381    4918 scope.go:122] "RemoveContainer" containerID="0af9ed300413693698053791469002e6c63e4f0a0fa12c342bdddc785097c1db"
Oct 30 21:45:24.232650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.232599    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"]
Oct 30 21:45:24.265178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.265145    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-b30a42-29889945-ttb9j"]
Oct 30 21:45:24.315453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.315356    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"]
Oct 30 21:45:24.350097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.350007    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"
Oct 30 21:45:24.483103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.482963    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c42b67a-e483-4511-9be1-005d53f02bdc-kube-api-access-98z5g\" (UniqueName: \"kubernetes.io/projected/2c42b67a-e483-4511-9be1-005d53f02bdc-kube-api-access-98z5g\") pod \"2c42b67a-e483-4511-9be1-005d53f02bdc\" (UID: \"2c42b67a-e483-4511-9be1-005d53f02bdc\") "
Oct 30 21:45:24.484136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.484106    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c42b67a-e483-4511-9be1-005d53f02bdc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c42b67a-e483-4511-9be1-005d53f02bdc-maas-api-service-ca\") pod \"2c42b67a-e483-4511-9be1-005d53f02bdc\" (UID: \"2c42b67a-e483-4511-9be1-005d53f02bdc\") "
Oct 30 21:45:24.489248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.485511    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c42b67a-e483-4511-9be1-005d53f02bdc-maas-api-service-ca" pod "2c42b67a-e483-4511-9be1-005d53f02bdc" (UID: "2c42b67a-e483-4511-9be1-005d53f02bdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:24.490609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.490581    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c42b67a-e483-4511-9be1-005d53f02bdc-kube-api-access-98z5g" pod "2c42b67a-e483-4511-9be1-005d53f02bdc" (UID: "2c42b67a-e483-4511-9be1-005d53f02bdc"). InnerVolumeSpecName "kube-api-access-98z5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:24.529724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.529697    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"
Oct 30 21:45:24.586981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.586907    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98z5g\" (UniqueName: \"kubernetes.io/projected/2c42b67a-e483-4511-9be1-005d53f02bdc-kube-api-access-98z5g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:24.586981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.586932    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c42b67a-e483-4511-9be1-005d53f02bdc-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:24.599422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.599371    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"
Oct 30 21:45:24.644606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.644582    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"
Oct 30 21:45:24.687861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.687830    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-kube-api-access-kdqp6\" (UniqueName: \"kubernetes.io/projected/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-kube-api-access-kdqp6\") pod \"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af\" (UID: \"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af\") "
Oct 30 21:45:24.688064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.687913    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/296ad3ea-d106-480c-b05c-2c44452d39ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/296ad3ea-d106-480c-b05c-2c44452d39ca-maas-api-service-ca\") pod \"296ad3ea-d106-480c-b05c-2c44452d39ca\" (UID: \"296ad3ea-d106-480c-b05c-2c44452d39ca\") "
Oct 30 21:45:24.688064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.688026    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-maas-api-service-ca\") pod \"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af\" (UID: \"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af\") "
Oct 30 21:45:24.688186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.688076    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296ad3ea-d106-480c-b05c-2c44452d39ca-kube-api-access-vdpl9\" (UniqueName: \"kubernetes.io/projected/296ad3ea-d106-480c-b05c-2c44452d39ca-kube-api-access-vdpl9\") pod \"296ad3ea-d106-480c-b05c-2c44452d39ca\" (UID: \"296ad3ea-d106-480c-b05c-2c44452d39ca\") "
Oct 30 21:45:24.688596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.688567    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296ad3ea-d106-480c-b05c-2c44452d39ca-maas-api-service-ca" pod "296ad3ea-d106-480c-b05c-2c44452d39ca" (UID: "296ad3ea-d106-480c-b05c-2c44452d39ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:24.688767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.688750    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/296ad3ea-d106-480c-b05c-2c44452d39ca-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:24.688926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.688891    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-maas-api-service-ca" pod "e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" (UID: "e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:24.690923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.690893    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-kube-api-access-kdqp6" pod "e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" (UID: "e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af"). InnerVolumeSpecName "kube-api-access-kdqp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:24.691014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.690978    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296ad3ea-d106-480c-b05c-2c44452d39ca-kube-api-access-vdpl9" pod "296ad3ea-d106-480c-b05c-2c44452d39ca" (UID: "296ad3ea-d106-480c-b05c-2c44452d39ca"). InnerVolumeSpecName "kube-api-access-vdpl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:24.756031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.755993    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9" podUID="ad4fcb54-9c19-4c50-b5fb-c07455a18720" containerName="payload-processing" containerID="cri-o://084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15" gracePeriod=25
Oct 30 21:45:24.789568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.789544    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b80735b-3681-4774-b4a3-821f37748327-kube-api-access-8szsj\" (UniqueName: \"kubernetes.io/projected/4b80735b-3681-4774-b4a3-821f37748327-kube-api-access-8szsj\") pod \"4b80735b-3681-4774-b4a3-821f37748327\" (UID: \"4b80735b-3681-4774-b4a3-821f37748327\") "
Oct 30 21:45:24.789819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.789800    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b80735b-3681-4774-b4a3-821f37748327-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b80735b-3681-4774-b4a3-821f37748327-maas-api-service-ca\") pod \"4b80735b-3681-4774-b4a3-821f37748327\" (UID: \"4b80735b-3681-4774-b4a3-821f37748327\") "
Oct 30 21:45:24.790101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.790073    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b80735b-3681-4774-b4a3-821f37748327-maas-api-service-ca" pod "4b80735b-3681-4774-b4a3-821f37748327" (UID: "4b80735b-3681-4774-b4a3-821f37748327"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:24.790497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.790481    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:24.790551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.790500    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdpl9\" (UniqueName: \"kubernetes.io/projected/296ad3ea-d106-480c-b05c-2c44452d39ca-kube-api-access-vdpl9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:24.790551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.790510    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b80735b-3681-4774-b4a3-821f37748327-maas-api-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:24.790551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.790519    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdqp6\" (UniqueName: \"kubernetes.io/projected/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af-kube-api-access-kdqp6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:24.791705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.791689    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b80735b-3681-4774-b4a3-821f37748327-kube-api-access-8szsj" pod "4b80735b-3681-4774-b4a3-821f37748327" (UID: "4b80735b-3681-4774-b4a3-821f37748327"). InnerVolumeSpecName "kube-api-access-8szsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:24.891963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:24.891930    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8szsj\" (UniqueName: \"kubernetes.io/projected/4b80735b-3681-4774-b4a3-821f37748327-kube-api-access-8szsj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:25.037465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.037436    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:45:25.085587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.085553    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06cfd24a-7880-40a4-98ed-0c9326491795" path="/var/lib/kubelet/pods/06cfd24a-7880-40a4-98ed-0c9326491795/volumes"
Oct 30 21:45:25.086141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.086123    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc" path="/var/lib/kubelet/pods/409b4a57-a05d-49c5-92e0-58a4b1712afc/volumes"
Oct 30 21:45:25.086624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.086606    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e008fe-d6b5-4328-a70d-188bdb928a4d" path="/var/lib/kubelet/pods/57e008fe-d6b5-4328-a70d-188bdb928a4d/volumes"
Oct 30 21:45:25.087367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.087349    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a319997-3bb4-419a-a27b-23849ee7691b" path="/var/lib/kubelet/pods/5a319997-3bb4-419a-a27b-23849ee7691b/volumes"
Oct 30 21:45:25.087805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.087790    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2" path="/var/lib/kubelet/pods/6bba61f8-4422-4206-8723-40d9ac86a4c2/volumes"
Oct 30 21:45:25.088218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.088202    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83727964-4998-42ed-82b2-7ac611269201" path="/var/lib/kubelet/pods/83727964-4998-42ed-82b2-7ac611269201/volumes"
Oct 30 21:45:25.088935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.088918    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" path="/var/lib/kubelet/pods/ee00d03a-36a1-47fe-be99-0cf595abd3a5/volumes"
Oct 30 21:45:25.089322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.089306    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b82123-e7fd-41ff-9655-c2610e1973bf" path="/var/lib/kubelet/pods/f6b82123-e7fd-41ff-9655-c2610e1973bf/volumes"
Oct 30 21:45:25.096847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.095116    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-provider-credentials\") pod \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") "
Oct 30 21:45:25.096847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.095365    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-plugins-config-volume\") pod \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") "
Oct 30 21:45:25.096847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.095527    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-routing-overlay-volume\") pod \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") "
Oct 30 21:45:25.096847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.095738    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-kube-api-access-nzz2w\" (UniqueName: \"kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-kube-api-access-nzz2w\") pod \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\" (UID: \"ad4fcb54-9c19-4c50-b5fb-c07455a18720\") "
Oct 30 21:45:25.097521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.097472    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-routing-overlay-volume" pod "ad4fcb54-9c19-4c50-b5fb-c07455a18720" (UID: "ad4fcb54-9c19-4c50-b5fb-c07455a18720"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:25.098366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.098346    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-plugins-config-volume" pod "ad4fcb54-9c19-4c50-b5fb-c07455a18720" (UID: "ad4fcb54-9c19-4c50-b5fb-c07455a18720"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:25.101599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.101274    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-provider-credentials" pod "ad4fcb54-9c19-4c50-b5fb-c07455a18720" (UID: "ad4fcb54-9c19-4c50-b5fb-c07455a18720"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:25.101599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.101362    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-kube-api-access-nzz2w" pod "ad4fcb54-9c19-4c50-b5fb-c07455a18720" (UID: "ad4fcb54-9c19-4c50-b5fb-c07455a18720"). InnerVolumeSpecName "kube-api-access-nzz2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:25.136934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.136862    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"
Oct 30 21:45:25.136934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.136878    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs" event={"ID":"e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af","Type":"ContainerDied","Data":"22967884c93c500bb51b296e05b3e448ca08959dcd07852296798f6ccd5c0a8d"}
Oct 30 21:45:25.137135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.136948    4918 scope.go:122] "RemoveContainer" containerID="224685a1c9a7c7dbc103e95cc83d8c1288a2db692862e9bb170f32ef468ca801"
Oct 30 21:45:25.138927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.138905    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"
Oct 30 21:45:25.139052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.138923    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd" event={"ID":"2c42b67a-e483-4511-9be1-005d53f02bdc","Type":"ContainerDied","Data":"210a52bd90854dfd23dd162912cb7be29b1db48381ee8d451eab2abd5dd688b6"}
Oct 30 21:45:25.140638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.140612    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"
Oct 30 21:45:25.140778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.140611    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj" event={"ID":"4b80735b-3681-4774-b4a3-821f37748327","Type":"ContainerDied","Data":"1253e6a63f0265db42215c623307320c87fee93d72520375dcfc3e8308c742d9"}
Oct 30 21:45:25.142913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.142895    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"
Oct 30 21:45:25.143007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.142927    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6" event={"ID":"296ad3ea-d106-480c-b05c-2c44452d39ca","Type":"ContainerDied","Data":"4499772b60e0366408a6474fd3a09937f7e4c93f72fd5263047fd15801975027"}
Oct 30 21:45:25.145960 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.145935    4918 generic.go:356] "Generic (PLEG): container finished" podID="ad4fcb54-9c19-4c50-b5fb-c07455a18720" containerID="084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15" exitCode=0
Oct 30 21:45:25.146065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.145988    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"
Oct 30 21:45:25.146065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.146015    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9" event={"ID":"ad4fcb54-9c19-4c50-b5fb-c07455a18720","Type":"ContainerDied","Data":"084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15"}
Oct 30 21:45:25.146168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.146076    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9" event={"ID":"ad4fcb54-9c19-4c50-b5fb-c07455a18720","Type":"ContainerDied","Data":"809a156c7643ce4473dd196cfe7818af49c0aa632df0418807849c3df9109365"}
Oct 30 21:45:25.153581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.153561    4918 scope.go:122] "RemoveContainer" containerID="30e8f8884e5df451ac3741c23b686d9865948bed9c05e9e877b796336c057467"
Oct 30 21:45:25.166647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.166623    4918 scope.go:122] "RemoveContainer" containerID="9d0b3bf026e0379b3d3567f42ea2386f9db42b7bbba11e39a0c2360c1d5c26f2"
Oct 30 21:45:25.171431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.171363    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"]
Oct 30 21:45:25.177456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.177434    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3011c6-29889945-b6xvs"]
Oct 30 21:45:25.177798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.177778    4918 scope.go:122] "RemoveContainer" containerID="d1a40ec8ec1227318cb78bc091b22167bb5b9127213a803127799cac24ddc24d"
Oct 30 21:45:25.189632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.189615    4918 scope.go:122] "RemoveContainer" containerID="084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15"
Oct 30 21:45:25.203313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.203279    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"]
Oct 30 21:45:25.206207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.206177    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:25.206350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.206210    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ad4fcb54-9c19-4c50-b5fb-c07455a18720-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:25.206350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.206227    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzz2w\" (UniqueName: \"kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-kube-api-access-nzz2w\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:25.206350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.206240    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ad4fcb54-9c19-4c50-b5fb-c07455a18720-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:25.208411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.208365    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-904993-29889945-2qqtj"]
Oct 30 21:45:25.214889 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.214867    4918 scope.go:122] "RemoveContainer" containerID="084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15"
Oct 30 21:45:25.215284 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:25.215257    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15\": container with ID starting with 084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15 not found: ID does not exist" containerID="084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15"
Oct 30 21:45:25.215374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.215293    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15"} err="failed to get container status \"084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15\": rpc error: code = NotFound desc = could not find container \"084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15\": container with ID starting with 084bc0e77d85c2f072f67b5ab69902dc3a3adfb010ea0c24817e2f40ff64bf15 not found: ID does not exist"
Oct 30 21:45:25.218845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.218826    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"]
Oct 30 21:45:25.226167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.226144    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0ea1ea/payload-processing-external-model-e2e-extmod-b-0ea1ea-86ccxbmf9"]
Oct 30 21:45:25.238813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.238795    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"]
Oct 30 21:45:25.244561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.244543    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0ea1ea-29889945-ndxgd"]
Oct 30 21:45:25.257877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.257857    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"]
Oct 30 21:45:25.381175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:25.381139    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f9e5a7-29889945-hgmx6"]
Oct 30 21:45:27.083175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:27.083144    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="296ad3ea-d106-480c-b05c-2c44452d39ca" path="/var/lib/kubelet/pods/296ad3ea-d106-480c-b05c-2c44452d39ca/volumes"
Oct 30 21:45:27.083756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:27.083739    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c42b67a-e483-4511-9be1-005d53f02bdc" path="/var/lib/kubelet/pods/2c42b67a-e483-4511-9be1-005d53f02bdc/volumes"
Oct 30 21:45:27.084244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:27.084228    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b80735b-3681-4774-b4a3-821f37748327" path="/var/lib/kubelet/pods/4b80735b-3681-4774-b4a3-821f37748327/volumes"
Oct 30 21:45:27.085019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:27.085004    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4fcb54-9c19-4c50-b5fb-c07455a18720" path="/var/lib/kubelet/pods/ad4fcb54-9c19-4c50-b5fb-c07455a18720/volumes"
Oct 30 21:45:27.085452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:27.085439    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" path="/var/lib/kubelet/pods/e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af/volumes"
Oct 30 21:45:29.163517 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:29.163475    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:45:32.923416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:32.912100    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:45:33.330417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:33.329152    4918 scope.go:122] "RemoveContainer" containerID="7405659c94bbe938893f778859afb2390e6dbd0352145846cc821b6c57e195a7"
Oct 30 21:45:33.436989 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:33.436628    4918 scope.go:122] "RemoveContainer" containerID="cd9e3162c1d8492b997b0f7d616f05ba518f01bc477ba521b911559cd83f5aca"
Oct 30 21:45:33.494062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:33.493994    4918 scope.go:122] "RemoveContainer" containerID="88e71195d6559938922daa37c6525951177a6e35283f6ccafa6d1e4740bd0fcd"
Oct 30 21:45:33.513580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:33.513542    4918 scope.go:122] "RemoveContainer" containerID="16abb8bdd03f247ec46569dc4f813ec4a3645f941cf9cb641dbebdfacfa4041c"
Oct 30 21:45:33.537569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:45:33.537541    4918 scope.go:122] "RemoveContainer" containerID="973332ba009b96ecea2510c0343e179e2563ef7151814c36daabd6e8d25e59d7"
Oct 30 21:45:59.164700 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:45:59.164612    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:46:07.277149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.277109    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"]
Oct 30 21:46:07.277791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.277166    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c4f40c-mbz88"]
Oct 30 21:46:07.849202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.849166    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"]
Oct 30 21:46:07.850042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850025    4918 cpu_manager.go:414] "containerMap: removing container" podUID="296ad3ea-d106-480c-b05c-2c44452d39ca" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850043    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296ad3ea-d106-480c-b05c-2c44452d39ca" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850054    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c3712d27-9480-45ce-a089-caff86ddb3ec" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850059    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3712d27-9480-45ce-a089-caff86ddb3ec" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850071    4918 cpu_manager.go:414] "containerMap: removing container" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850076    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850082    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850089    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850099    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5a319997-3bb4-419a-a27b-23849ee7691b" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850106    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a319997-3bb4-419a-a27b-23849ee7691b" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850118    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850123    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850129    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850133    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850142    4918 cpu_manager.go:414] "containerMap: removing container" podUID="83727964-4998-42ed-82b2-7ac611269201" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850146    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83727964-4998-42ed-82b2-7ac611269201" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850154    4918 cpu_manager.go:414] "containerMap: removing container" podUID="57e008fe-d6b5-4328-a70d-188bdb928a4d" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850158    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e008fe-d6b5-4328-a70d-188bdb928a4d" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850165    4918 cpu_manager.go:414] "containerMap: removing container" podUID="83727964-4998-42ed-82b2-7ac611269201" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850169    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83727964-4998-42ed-82b2-7ac611269201" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850178    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850184    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850189    4918 cpu_manager.go:414] "containerMap: removing container" podUID="06cfd24a-7880-40a4-98ed-0c9326491795" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850194    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06cfd24a-7880-40a4-98ed-0c9326491795" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850207    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e65d8a21-107a-40ad-b8fc-967cb5008270" containerName="cleanup"
Oct 30 21:46:07.850201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850213    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65d8a21-107a-40ad-b8fc-967cb5008270" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850223    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5a319997-3bb4-419a-a27b-23849ee7691b" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850227    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a319997-3bb4-419a-a27b-23849ee7691b" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850247    4918 cpu_manager.go:414] "containerMap: removing container" podUID="83727964-4998-42ed-82b2-7ac611269201" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850252    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83727964-4998-42ed-82b2-7ac611269201" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850260    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ad4fcb54-9c19-4c50-b5fb-c07455a18720" containerName="payload-processing"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850264    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4fcb54-9c19-4c50-b5fb-c07455a18720" containerName="payload-processing"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850272    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f6b82123-e7fd-41ff-9655-c2610e1973bf" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850276    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b82123-e7fd-41ff-9655-c2610e1973bf" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850283    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850288    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850296    4918 cpu_manager.go:414] "containerMap: removing container" podUID="aedc6368-2835-4821-bf6c-9cb838f21611" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850300    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedc6368-2835-4821-bf6c-9cb838f21611" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850307    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2c42b67a-e483-4511-9be1-005d53f02bdc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850313    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c42b67a-e483-4511-9be1-005d53f02bdc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850331    4918 cpu_manager.go:414] "containerMap: removing container" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850336    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850342    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f6b82123-e7fd-41ff-9655-c2610e1973bf" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850346    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b82123-e7fd-41ff-9655-c2610e1973bf" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850352    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850356    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850365    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2c42b67a-e483-4511-9be1-005d53f02bdc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850369    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c42b67a-e483-4511-9be1-005d53f02bdc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850379    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e65d8a21-107a-40ad-b8fc-967cb5008270" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850383    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65d8a21-107a-40ad-b8fc-967cb5008270" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850407    4918 cpu_manager.go:414] "containerMap: removing container" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850414    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850421    4918 cpu_manager.go:414] "containerMap: removing container" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850425    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850431    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e65d8a21-107a-40ad-b8fc-967cb5008270" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850435    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65d8a21-107a-40ad-b8fc-967cb5008270" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850443    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a60f039f-582f-4178-916d-5bf0260ed803" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850447    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60f039f-582f-4178-916d-5bf0260ed803" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850453    4918 cpu_manager.go:414] "containerMap: removing container" podUID="57e008fe-d6b5-4328-a70d-188bdb928a4d" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850461    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e008fe-d6b5-4328-a70d-188bdb928a4d" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850466    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2c42b67a-e483-4511-9be1-005d53f02bdc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850471    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c42b67a-e483-4511-9be1-005d53f02bdc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850488    4918 cpu_manager.go:414] "containerMap: removing container" podUID="aedc6368-2835-4821-bf6c-9cb838f21611" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850493    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedc6368-2835-4821-bf6c-9cb838f21611" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850503    4918 cpu_manager.go:414] "containerMap: removing container" podUID="aedc6368-2835-4821-bf6c-9cb838f21611" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850508    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedc6368-2835-4821-bf6c-9cb838f21611" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850518    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4b80735b-3681-4774-b4a3-821f37748327" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850523    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b80735b-3681-4774-b4a3-821f37748327" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850529    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a60f039f-582f-4178-916d-5bf0260ed803" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850534    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60f039f-582f-4178-916d-5bf0260ed803" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850541    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a60f039f-582f-4178-916d-5bf0260ed803" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850545    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60f039f-582f-4178-916d-5bf0260ed803" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850552    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4b80735b-3681-4774-b4a3-821f37748327" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850556    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b80735b-3681-4774-b4a3-821f37748327" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850561    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c3712d27-9480-45ce-a089-caff86ddb3ec" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850565    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3712d27-9480-45ce-a089-caff86ddb3ec" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850572    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4b80735b-3681-4774-b4a3-821f37748327" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850576    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b80735b-3681-4774-b4a3-821f37748327" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850583    4918 cpu_manager.go:414] "containerMap: removing container" podUID="06cfd24a-7880-40a4-98ed-0c9326491795" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850587    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06cfd24a-7880-40a4-98ed-0c9326491795" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850594    4918 cpu_manager.go:414] "containerMap: removing container" podUID="296ad3ea-d106-480c-b05c-2c44452d39ca" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850598    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296ad3ea-d106-480c-b05c-2c44452d39ca" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850605    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850609    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850736    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a319997-3bb4-419a-a27b-23849ee7691b" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850745    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850753    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850758    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="83727964-4998-42ed-82b2-7ac611269201" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850766    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4fcb54-9c19-4c50-b5fb-c07455a18720" containerName="payload-processing"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850774    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b82123-e7fd-41ff-9655-c2610e1973bf" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850780    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850788    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedc6368-2835-4821-bf6c-9cb838f21611" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850794    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c42b67a-e483-4511-9be1-005d53f02bdc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850800    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850805    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b82123-e7fd-41ff-9655-c2610e1973bf" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850809    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850814    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="296ad3ea-d106-480c-b05c-2c44452d39ca" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850819    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c42b67a-e483-4511-9be1-005d53f02bdc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850837    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65d8a21-107a-40ad-b8fc-967cb5008270" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850842    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850847    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850853    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65d8a21-107a-40ad-b8fc-967cb5008270" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850858    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e008fe-d6b5-4328-a70d-188bdb928a4d" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850862    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60f039f-582f-4178-916d-5bf0260ed803" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850866    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e008fe-d6b5-4328-a70d-188bdb928a4d" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850871    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c42b67a-e483-4511-9be1-005d53f02bdc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850880    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850887    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedc6368-2835-4821-bf6c-9cb838f21611" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850892    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60f039f-582f-4178-916d-5bf0260ed803" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850897    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3712d27-9480-45ce-a089-caff86ddb3ec" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850903    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="06cfd24a-7880-40a4-98ed-0c9326491795" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850909    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedc6368-2835-4821-bf6c-9cb838f21611" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850913    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b80735b-3681-4774-b4a3-821f37748327" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850918    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b80735b-3681-4774-b4a3-821f37748327" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850922    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3712d27-9480-45ce-a089-caff86ddb3ec" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850928    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b80735b-3681-4774-b4a3-821f37748327" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850934    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="06cfd24a-7880-40a4-98ed-0c9326491795" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850940    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850945    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="296ad3ea-d106-480c-b05c-2c44452d39ca" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850950    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3712d27-9480-45ce-a089-caff86ddb3ec" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850954    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850960    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850966    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a319997-3bb4-419a-a27b-23849ee7691b" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850974    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="83727964-4998-42ed-82b2-7ac611269201" containerName="cleanup"
Oct 30 21:46:07.851059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.850980    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2" containerName="cleanup"
Oct 30 21:46:07.855825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.855124    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:07.860231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.860213    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:46:07.860554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.860351    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:46:07.860662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.860372    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:46:07.860662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.860448    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:46:07.862116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.862096    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"]
Oct 30 21:46:07.898120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.898087    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5811b415-schj7\" (UID: \"c13caba9-8805-41c5-b3c4-f1053933c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:07.898318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.898260    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72zs8\" (UniqueName: \"kubernetes.io/projected/c13caba9-8805-41c5-b3c4-f1053933c608-kube-api-access-72zs8\") pod \"maas-api-revoke-sub-5811b415-schj7\" (UID: \"c13caba9-8805-41c5-b3c4-f1053933c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:07.898460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:07.898377    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5811b415-schj7\" (UID: \"c13caba9-8805-41c5-b3c4-f1053933c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:08.000083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:08.000046    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5811b415-schj7\" (UID: \"c13caba9-8805-41c5-b3c4-f1053933c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:08.000334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:08.000315    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5811b415-schj7\" (UID: \"c13caba9-8805-41c5-b3c4-f1053933c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:08.000515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:08.000498    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72zs8\" (UniqueName: \"kubernetes.io/projected/c13caba9-8805-41c5-b3c4-f1053933c608-kube-api-access-72zs8\") pod \"maas-api-revoke-sub-5811b415-schj7\" (UID: \"c13caba9-8805-41c5-b3c4-f1053933c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:08.000850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:08.000833    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5811b415-schj7\" (UID: \"c13caba9-8805-41c5-b3c4-f1053933c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:08.002962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:08.002931    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5811b415-schj7\" (UID: \"c13caba9-8805-41c5-b3c4-f1053933c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:08.009254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:08.009233    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72zs8\" (UniqueName: \"kubernetes.io/projected/c13caba9-8805-41c5-b3c4-f1053933c608-kube-api-access-72zs8\") pod \"maas-api-revoke-sub-5811b415-schj7\" (UID: \"c13caba9-8805-41c5-b3c4-f1053933c608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:08.183482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:08.183258    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:08.519151 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:46:08.519124    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13caba9_8805_41c5_b3c4_f1053933c608.slice/crio-cdeff56e9c47e8acde12a06993fc4b8fab0efb02d700665efd49563929fdd367 WatchSource:0}: Error finding container cdeff56e9c47e8acde12a06993fc4b8fab0efb02d700665efd49563929fdd367: Status 404 returned error can't find the container with id cdeff56e9c47e8acde12a06993fc4b8fab0efb02d700665efd49563929fdd367
Oct 30 21:46:08.519540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:08.519379    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"]
Oct 30 21:46:09.082865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:09.082832    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3" path="/var/lib/kubelet/pods/49a7cd32-62c7-4d7e-a67f-eb8da6cc98c3/volumes"
Oct 30 21:46:09.302115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:09.302076    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7" event={"ID":"c13caba9-8805-41c5-b3c4-f1053933c608","Type":"ContainerStarted","Data":"349a191a77241ddab187b646e28dcc2f03bacb2d55ad249e26cfafdaac535191"}
Oct 30 21:46:09.302307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:09.302141    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7" event={"ID":"c13caba9-8805-41c5-b3c4-f1053933c608","Type":"ContainerStarted","Data":"cdeff56e9c47e8acde12a06993fc4b8fab0efb02d700665efd49563929fdd367"}
Oct 30 21:46:09.324125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:09.324076    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7" podStartSLOduration=2.324063477 podStartE2EDuration="2.324063477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:09.32111339 +0000 UTC m=+2682.367672560" watchObservedRunningTime="2026-10-30 21:46:09.324063477 +0000 UTC m=+2682.370622666"
Oct 30 21:46:10.312597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:10.312556    4918 generic.go:356] "Generic (PLEG): container finished" podID="c13caba9-8805-41c5-b3c4-f1053933c608" containerID="349a191a77241ddab187b646e28dcc2f03bacb2d55ad249e26cfafdaac535191" exitCode=0
Oct 30 21:46:10.313020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:10.312633    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7" event={"ID":"c13caba9-8805-41c5-b3c4-f1053933c608","Type":"ContainerDied","Data":"349a191a77241ddab187b646e28dcc2f03bacb2d55ad249e26cfafdaac535191"}
Oct 30 21:46:11.529367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:11.529341    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:11.660381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:11.647710    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13caba9-8805-41c5-b3c4-f1053933c608-kube-api-access-72zs8\" (UniqueName: \"kubernetes.io/projected/c13caba9-8805-41c5-b3c4-f1053933c608-kube-api-access-72zs8\") pod \"c13caba9-8805-41c5-b3c4-f1053933c608\" (UID: \"c13caba9-8805-41c5-b3c4-f1053933c608\") "
Oct 30 21:46:11.674203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:11.660879    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-xks-ca\") pod \"c13caba9-8805-41c5-b3c4-f1053933c608\" (UID: \"c13caba9-8805-41c5-b3c4-f1053933c608\") "
Oct 30 21:46:11.675383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:11.672546    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-xks-ca" pod "c13caba9-8805-41c5-b3c4-f1053933c608" (UID: "c13caba9-8805-41c5-b3c4-f1053933c608"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:11.688474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:11.681562    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-openshift-ca" pod "c13caba9-8805-41c5-b3c4-f1053933c608" (UID: "c13caba9-8805-41c5-b3c4-f1053933c608"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:11.688691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:11.677509    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-openshift-ca\") pod \"c13caba9-8805-41c5-b3c4-f1053933c608\" (UID: \"c13caba9-8805-41c5-b3c4-f1053933c608\") "
Oct 30 21:46:11.701884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:11.695935    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13caba9-8805-41c5-b3c4-f1053933c608-kube-api-access-72zs8" pod "c13caba9-8805-41c5-b3c4-f1053933c608" (UID: "c13caba9-8805-41c5-b3c4-f1053933c608"). InnerVolumeSpecName "kube-api-access-72zs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:11.748437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:11.738088    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72zs8\" (UniqueName: \"kubernetes.io/projected/c13caba9-8805-41c5-b3c4-f1053933c608-kube-api-access-72zs8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:11.748437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:11.738127    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:11.748437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:11.738144    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13caba9-8805-41c5-b3c4-f1053933c608-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:12.334835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:12.334804    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"
Oct 30 21:46:12.335022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:12.334848    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7" event={"ID":"c13caba9-8805-41c5-b3c4-f1053933c608","Type":"ContainerDied","Data":"cdeff56e9c47e8acde12a06993fc4b8fab0efb02d700665efd49563929fdd367"}
Oct 30 21:46:12.335022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:12.334910    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdeff56e9c47e8acde12a06993fc4b8fab0efb02d700665efd49563929fdd367"
Oct 30 21:46:17.270800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.270760    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"]
Oct 30 21:46:17.271911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.271887    4918 cpu_manager.go:414] "containerMap: removing container" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc" containerName="cleanup"
Oct 30 21:46:17.271911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.271908    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.271939    4918 cpu_manager.go:414] "containerMap: removing container" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.271946    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a3898f-6fd6-4db6-811b-48fdef8f81ba" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.271974    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c13caba9-8805-41c5-b3c4-f1053933c608" containerName="revoke-keys"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.271984    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13caba9-8805-41c5-b3c4-f1053933c608" containerName="revoke-keys"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272009    4918 cpu_manager.go:414] "containerMap: removing container" podUID="296ad3ea-d106-480c-b05c-2c44452d39ca" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272016    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296ad3ea-d106-480c-b05c-2c44452d39ca" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272028    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c3712d27-9480-45ce-a089-caff86ddb3ec" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272034    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3712d27-9480-45ce-a089-caff86ddb3ec" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272046    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272052    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ca05f8-a080-4b41-a3ab-5e8fdbc3d3af" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272061    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5a319997-3bb4-419a-a27b-23849ee7691b" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272068    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a319997-3bb4-419a-a27b-23849ee7691b" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272088    4918 cpu_manager.go:414] "containerMap: removing container" podUID="57e008fe-d6b5-4328-a70d-188bdb928a4d" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272095    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e008fe-d6b5-4328-a70d-188bdb928a4d" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272110    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272117    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2" containerName="cleanup"
Oct 30 21:46:17.272121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272125    4918 cpu_manager.go:414] "containerMap: removing container" podUID="06cfd24a-7880-40a4-98ed-0c9326491795" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272133    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06cfd24a-7880-40a4-98ed-0c9326491795" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272175    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f6b82123-e7fd-41ff-9655-c2610e1973bf" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272185    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b82123-e7fd-41ff-9655-c2610e1973bf" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272482    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="06cfd24a-7880-40a4-98ed-0c9326491795" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272504    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="296ad3ea-d106-480c-b05c-2c44452d39ca" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272514    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a319997-3bb4-419a-a27b-23849ee7691b" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272527    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bba61f8-4422-4206-8723-40d9ac86a4c2" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272556    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="409b4a57-a05d-49c5-92e0-58a4b1712afc" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272567    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="83727964-4998-42ed-82b2-7ac611269201" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272582    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b82123-e7fd-41ff-9655-c2610e1973bf" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272599    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee00d03a-36a1-47fe-be99-0cf595abd3a5" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272627    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65d8a21-107a-40ad-b8fc-967cb5008270" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272639    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e008fe-d6b5-4328-a70d-188bdb928a4d" containerName="cleanup"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272650    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13caba9-8805-41c5-b3c4-f1053933c608" containerName="revoke-keys"
Oct 30 21:46:17.273013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.272667    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60f039f-582f-4178-916d-5bf0260ed803" containerName="cleanup"
Oct 30 21:46:17.278491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.278468    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"
Oct 30 21:46:17.285726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.285681    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"]
Oct 30 21:46:17.286662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.286616    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-570f01\""
Oct 30 21:46:17.286794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.286691    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-570f01-dockercfg-t6pfr\""
Oct 30 21:46:17.292009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.291990    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"
Oct 30 21:46:17.293216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.293197    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"]
Oct 30 21:46:17.634702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.500503    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"]
Oct 30 21:46:17.634702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.511078    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swxnq\" (UniqueName: \"kubernetes.io/projected/45c3c165-44f0-4b29-ba24-01c7e16a0e22-kube-api-access-swxnq\") pod \"payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95\" (UID: \"45c3c165-44f0-4b29-ba24-01c7e16a0e22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"
Oct 30 21:46:17.634702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.511150    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd30e979-c50a-4be3-9ed5-edd99eef7c67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg\" (UID: \"cd30e979-c50a-4be3-9ed5-edd99eef7c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"
Oct 30 21:46:17.634702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.511233    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c3c165-44f0-4b29-ba24-01c7e16a0e22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95\" (UID: \"45c3c165-44f0-4b29-ba24-01c7e16a0e22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"
Oct 30 21:46:17.634702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.511358    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpkkp\" (UniqueName: \"kubernetes.io/projected/cd30e979-c50a-4be3-9ed5-edd99eef7c67-kube-api-access-kpkkp\") pod \"payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg\" (UID: \"cd30e979-c50a-4be3-9ed5-edd99eef7c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"
Oct 30 21:46:17.634702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.616286    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd30e979-c50a-4be3-9ed5-edd99eef7c67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg\" (UID: \"cd30e979-c50a-4be3-9ed5-edd99eef7c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"
Oct 30 21:46:17.634702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.616415    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c3c165-44f0-4b29-ba24-01c7e16a0e22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95\" (UID: \"45c3c165-44f0-4b29-ba24-01c7e16a0e22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"
Oct 30 21:46:17.634702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.616551    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpkkp\" (UniqueName: \"kubernetes.io/projected/cd30e979-c50a-4be3-9ed5-edd99eef7c67-kube-api-access-kpkkp\") pod \"payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg\" (UID: \"cd30e979-c50a-4be3-9ed5-edd99eef7c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"
Oct 30 21:46:17.634702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.616839    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxnq\" (UniqueName: \"kubernetes.io/projected/45c3c165-44f0-4b29-ba24-01c7e16a0e22-kube-api-access-swxnq\") pod \"payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95\" (UID: \"45c3c165-44f0-4b29-ba24-01c7e16a0e22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"
Oct 30 21:46:17.688052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.688018    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c3c165-44f0-4b29-ba24-01c7e16a0e22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95\" (UID: \"45c3c165-44f0-4b29-ba24-01c7e16a0e22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"
Oct 30 21:46:17.689712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.689674    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd30e979-c50a-4be3-9ed5-edd99eef7c67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg\" (UID: \"cd30e979-c50a-4be3-9ed5-edd99eef7c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"
Oct 30 21:46:17.725910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.725504    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpkkp\" (UniqueName: \"kubernetes.io/projected/cd30e979-c50a-4be3-9ed5-edd99eef7c67-kube-api-access-kpkkp\") pod \"payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg\" (UID: \"cd30e979-c50a-4be3-9ed5-edd99eef7c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"
Oct 30 21:46:17.734219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.733988    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxnq\" (UniqueName: \"kubernetes.io/projected/45c3c165-44f0-4b29-ba24-01c7e16a0e22-kube-api-access-swxnq\") pod \"payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95\" (UID: \"45c3c165-44f0-4b29-ba24-01c7e16a0e22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"
Oct 30 21:46:17.777747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.777711    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"
Oct 30 21:46:17.899225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:17.899137    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"
Oct 30 21:46:18.025843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.025806    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"]
Oct 30 21:46:18.031118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.031087    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cfe707f-l798j"]
Oct 30 21:46:18.079818 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:46:18.079789    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c3c165_44f0_4b29_ba24_01c7e16a0e22.slice/crio-2c1c86d7482cc5d874e589cbd7d51124d0c46b818d1c30a187bbfc2c98ef4706 WatchSource:0}: Error finding container 2c1c86d7482cc5d874e589cbd7d51124d0c46b818d1c30a187bbfc2c98ef4706: Status 404 returned error can't find the container with id 2c1c86d7482cc5d874e589cbd7d51124d0c46b818d1c30a187bbfc2c98ef4706
Oct 30 21:46:18.080840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.080819    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"]
Oct 30 21:46:18.261072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.261036    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"]
Oct 30 21:46:18.265439 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:46:18.265414    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd30e979_c50a_4be3_9ed5_edd99eef7c67.slice/crio-5817bf0cef0cd0d6268b019b7eb5ae5e087023dec73f87a40ee49e6c1a9b6e8f WatchSource:0}: Error finding container 5817bf0cef0cd0d6268b019b7eb5ae5e087023dec73f87a40ee49e6c1a9b6e8f: Status 404 returned error can't find the container with id 5817bf0cef0cd0d6268b019b7eb5ae5e087023dec73f87a40ee49e6c1a9b6e8f
Oct 30 21:46:18.419471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.419381    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"]
Oct 30 21:46:18.424622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.424600    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:18.433024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.432903    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:46:18.433332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.433309    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:46:18.434861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.434830    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:46:18.442023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.441998    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:46:18.442528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.442491    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"]
Oct 30 21:46:18.543592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.543558    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17a9816d-nv7tr\" (UID: \"687e7406-3b53-4568-9cdd-2ba41287568a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:18.543764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.543671    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brkr\" (UniqueName: \"kubernetes.io/projected/687e7406-3b53-4568-9cdd-2ba41287568a-kube-api-access-9brkr\") pod \"maas-api-revoke-sub-17a9816d-nv7tr\" (UID: \"687e7406-3b53-4568-9cdd-2ba41287568a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:18.543764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.543731    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17a9816d-nv7tr\" (UID: \"687e7406-3b53-4568-9cdd-2ba41287568a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:18.547165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.547136    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg" event={"ID":"cd30e979-c50a-4be3-9ed5-edd99eef7c67","Type":"ContainerStarted","Data":"dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b"}
Oct 30 21:46:18.547298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.547192    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg" event={"ID":"cd30e979-c50a-4be3-9ed5-edd99eef7c67","Type":"ContainerStarted","Data":"5817bf0cef0cd0d6268b019b7eb5ae5e087023dec73f87a40ee49e6c1a9b6e8f"}
Oct 30 21:46:18.547298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.547229    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"
Oct 30 21:46:18.549045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.549021    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95" event={"ID":"45c3c165-44f0-4b29-ba24-01c7e16a0e22","Type":"ContainerStarted","Data":"6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea"}
Oct 30 21:46:18.549284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.549061    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95" event={"ID":"45c3c165-44f0-4b29-ba24-01c7e16a0e22","Type":"ContainerStarted","Data":"2c1c86d7482cc5d874e589cbd7d51124d0c46b818d1c30a187bbfc2c98ef4706"}
Oct 30 21:46:18.549284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.549160    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"
Oct 30 21:46:18.569772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.569733    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg" podStartSLOduration=1.5697213049999998 podStartE2EDuration="1.569721305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:18.566923534 +0000 UTC m=+2691.613482692" watchObservedRunningTime="2026-10-30 21:46:18.569721305 +0000 UTC m=+2691.616280449"
Oct 30 21:46:18.587896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.587853    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95" podStartSLOduration=1.5878404320000001 podStartE2EDuration="1.587840432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:18.585884275 +0000 UTC m=+2691.632443423" watchObservedRunningTime="2026-10-30 21:46:18.587840432 +0000 UTC m=+2691.634399577"
Oct 30 21:46:18.645512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.645478    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17a9816d-nv7tr\" (UID: \"687e7406-3b53-4568-9cdd-2ba41287568a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:18.645642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.645535    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brkr\" (UniqueName: \"kubernetes.io/projected/687e7406-3b53-4568-9cdd-2ba41287568a-kube-api-access-9brkr\") pod \"maas-api-revoke-sub-17a9816d-nv7tr\" (UID: \"687e7406-3b53-4568-9cdd-2ba41287568a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:18.645642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.645560    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17a9816d-nv7tr\" (UID: \"687e7406-3b53-4568-9cdd-2ba41287568a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:18.646218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.646193    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17a9816d-nv7tr\" (UID: \"687e7406-3b53-4568-9cdd-2ba41287568a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:18.648109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.648092    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17a9816d-nv7tr\" (UID: \"687e7406-3b53-4568-9cdd-2ba41287568a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:18.654671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.654649    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brkr\" (UniqueName: \"kubernetes.io/projected/687e7406-3b53-4568-9cdd-2ba41287568a-kube-api-access-9brkr\") pod \"maas-api-revoke-sub-17a9816d-nv7tr\" (UID: \"687e7406-3b53-4568-9cdd-2ba41287568a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:18.782244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:18.767035    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:19.085150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:19.085041    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3491ef-8d83-4636-8f33-9e7cbbd00afe" path="/var/lib/kubelet/pods/cb3491ef-8d83-4636-8f33-9e7cbbd00afe/volumes"
Oct 30 21:46:19.101910 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:46:19.101876    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687e7406_3b53_4568_9cdd_2ba41287568a.slice/crio-3196693f2782c47dd8ceccc1ae94f54ff8411c6c40e8fa119485ffdeab902ba1 WatchSource:0}: Error finding container 3196693f2782c47dd8ceccc1ae94f54ff8411c6c40e8fa119485ffdeab902ba1: Status 404 returned error can't find the container with id 3196693f2782c47dd8ceccc1ae94f54ff8411c6c40e8fa119485ffdeab902ba1
Oct 30 21:46:19.103229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:19.103207    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"]
Oct 30 21:46:19.564156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:19.564117    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr" event={"ID":"687e7406-3b53-4568-9cdd-2ba41287568a","Type":"ContainerStarted","Data":"f70e06a5cf82e46e5653f0b1e001af141339b20aed6f70df061db10bf698004a"}
Oct 30 21:46:19.564697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:19.564184    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr" event={"ID":"687e7406-3b53-4568-9cdd-2ba41287568a","Type":"ContainerStarted","Data":"3196693f2782c47dd8ceccc1ae94f54ff8411c6c40e8fa119485ffdeab902ba1"}
Oct 30 21:46:19.639437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:19.636483    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr" podStartSLOduration=1.6364624060000001 podStartE2EDuration="1.636462406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:19.607788193 +0000 UTC m=+2692.654347339" watchObservedRunningTime="2026-10-30 21:46:19.636462406 +0000 UTC m=+2692.683021552"
Oct 30 21:46:20.574734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:20.574697    4918 generic.go:356] "Generic (PLEG): container finished" podID="687e7406-3b53-4568-9cdd-2ba41287568a" containerID="f70e06a5cf82e46e5653f0b1e001af141339b20aed6f70df061db10bf698004a" exitCode=0
Oct 30 21:46:20.575204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:20.574753    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr" event={"ID":"687e7406-3b53-4568-9cdd-2ba41287568a","Type":"ContainerDied","Data":"f70e06a5cf82e46e5653f0b1e001af141339b20aed6f70df061db10bf698004a"}
Oct 30 21:46:21.772514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:21.772488    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:21.888401 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:21.888310    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-xks-ca\") pod \"687e7406-3b53-4568-9cdd-2ba41287568a\" (UID: \"687e7406-3b53-4568-9cdd-2ba41287568a\") "
Oct 30 21:46:21.888558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:21.888481    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/687e7406-3b53-4568-9cdd-2ba41287568a-kube-api-access-9brkr\" (UniqueName: \"kubernetes.io/projected/687e7406-3b53-4568-9cdd-2ba41287568a-kube-api-access-9brkr\") pod \"687e7406-3b53-4568-9cdd-2ba41287568a\" (UID: \"687e7406-3b53-4568-9cdd-2ba41287568a\") "
Oct 30 21:46:21.888711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:21.888693    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-openshift-ca\") pod \"687e7406-3b53-4568-9cdd-2ba41287568a\" (UID: \"687e7406-3b53-4568-9cdd-2ba41287568a\") "
Oct 30 21:46:21.889091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:21.889060    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-openshift-ca" pod "687e7406-3b53-4568-9cdd-2ba41287568a" (UID: "687e7406-3b53-4568-9cdd-2ba41287568a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:21.889908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:21.889884    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:21.890968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:21.890945    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687e7406-3b53-4568-9cdd-2ba41287568a-kube-api-access-9brkr" pod "687e7406-3b53-4568-9cdd-2ba41287568a" (UID: "687e7406-3b53-4568-9cdd-2ba41287568a"). InnerVolumeSpecName "kube-api-access-9brkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:21.891184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:21.891168    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-xks-ca" pod "687e7406-3b53-4568-9cdd-2ba41287568a" (UID: "687e7406-3b53-4568-9cdd-2ba41287568a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:22.033417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.032997    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687e7406-3b53-4568-9cdd-2ba41287568a-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:22.033417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.033085    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brkr\" (UniqueName: \"kubernetes.io/projected/687e7406-3b53-4568-9cdd-2ba41287568a-kube-api-access-9brkr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:22.592729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.592697    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr" event={"ID":"687e7406-3b53-4568-9cdd-2ba41287568a","Type":"ContainerDied","Data":"3196693f2782c47dd8ceccc1ae94f54ff8411c6c40e8fa119485ffdeab902ba1"}
Oct 30 21:46:22.592930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.592753    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3196693f2782c47dd8ceccc1ae94f54ff8411c6c40e8fa119485ffdeab902ba1"
Oct 30 21:46:22.592930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.592706    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"
Oct 30 21:46:22.953165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.953087    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"]
Oct 30 21:46:22.954004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.953988    4918 cpu_manager.go:414] "containerMap: removing container" podUID="687e7406-3b53-4568-9cdd-2ba41287568a" containerName="revoke-keys"
Oct 30 21:46:22.954004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.954006    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="687e7406-3b53-4568-9cdd-2ba41287568a" containerName="revoke-keys"
Oct 30 21:46:22.954157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.954148    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="687e7406-3b53-4568-9cdd-2ba41287568a" containerName="revoke-keys"
Oct 30 21:46:22.958361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.958335    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:22.964042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.964007    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:46:22.964190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.964065    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:46:22.964792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.964757    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:46:22.965204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.965182    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:46:22.966125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:22.966104    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"]
Oct 30 21:46:23.053954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.053925    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef07f59b-h6sj4\" (UID: \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:23.054139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.053987    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk72c\" (UniqueName: \"kubernetes.io/projected/f74c7e5b-34f0-480e-8f2a-63b406788ed5-kube-api-access-lk72c\") pod \"maas-api-revoke-sub-ef07f59b-h6sj4\" (UID: \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:23.054139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.054111    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef07f59b-h6sj4\" (UID: \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:23.156063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.156025    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef07f59b-h6sj4\" (UID: \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:23.156245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.156149    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk72c\" (UniqueName: \"kubernetes.io/projected/f74c7e5b-34f0-480e-8f2a-63b406788ed5-kube-api-access-lk72c\") pod \"maas-api-revoke-sub-ef07f59b-h6sj4\" (UID: \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:23.156343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.156260    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef07f59b-h6sj4\" (UID: \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:23.156719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.156695    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef07f59b-h6sj4\" (UID: \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:23.158788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.158767    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef07f59b-h6sj4\" (UID: \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:23.164378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.164360    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk72c\" (UniqueName: \"kubernetes.io/projected/f74c7e5b-34f0-480e-8f2a-63b406788ed5-kube-api-access-lk72c\") pod \"maas-api-revoke-sub-ef07f59b-h6sj4\" (UID: \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:23.277222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.277190    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:23.583595 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:46:23.583560    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74c7e5b_34f0_480e_8f2a_63b406788ed5.slice/crio-eef7f2be2840b8440e2fdfa7707b010cbc353626d899e768fdf31cb2a179fc67 WatchSource:0}: Error finding container eef7f2be2840b8440e2fdfa7707b010cbc353626d899e768fdf31cb2a179fc67: Status 404 returned error can't find the container with id eef7f2be2840b8440e2fdfa7707b010cbc353626d899e768fdf31cb2a179fc67
Oct 30 21:46:23.584414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.584375    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"]
Oct 30 21:46:23.604240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.604211    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4" event={"ID":"f74c7e5b-34f0-480e-8f2a-63b406788ed5","Type":"ContainerStarted","Data":"eef7f2be2840b8440e2fdfa7707b010cbc353626d899e768fdf31cb2a179fc67"}
Oct 30 21:46:23.845641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.845569    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"]
Oct 30 21:46:23.850533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.850505    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:23.861454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.861427    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"]
Oct 30 21:46:23.870427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.870381    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9232cc20-jxcr7\" (UID: \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:23.870564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.870448    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqdmt\" (UniqueName: \"kubernetes.io/projected/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-kube-api-access-lqdmt\") pod \"maas-api-revoke-sub-9232cc20-jxcr7\" (UID: \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:23.870658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.870643    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9232cc20-jxcr7\" (UID: \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:23.972525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.972488    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9232cc20-jxcr7\" (UID: \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:23.972525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.972527    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqdmt\" (UniqueName: \"kubernetes.io/projected/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-kube-api-access-lqdmt\") pod \"maas-api-revoke-sub-9232cc20-jxcr7\" (UID: \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:23.973104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.972622    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9232cc20-jxcr7\" (UID: \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:23.973308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.973288    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9232cc20-jxcr7\" (UID: \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:23.975296 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.975276    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9232cc20-jxcr7\" (UID: \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:23.983183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:23.983157    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqdmt\" (UniqueName: \"kubernetes.io/projected/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-kube-api-access-lqdmt\") pod \"maas-api-revoke-sub-9232cc20-jxcr7\" (UID: \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:24.169303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:24.169218    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:24.587659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:24.587621    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"]
Oct 30 21:46:24.621717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:24.621673    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4" event={"ID":"f74c7e5b-34f0-480e-8f2a-63b406788ed5","Type":"ContainerStarted","Data":"bfddf001ae2a0912506c402aa103a328a2d45336e12174bf41a9ec549b0cdc10"}
Oct 30 21:46:24.623782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:24.623749    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7" event={"ID":"08cf8023-9260-4e4a-9aa3-c801e7a6f85b","Type":"ContainerStarted","Data":"8132fafc0c1186b35147a2161446d1fb032b83eba3ae3285c46278f75c7b81de"}
Oct 30 21:46:25.633829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:25.633786    4918 generic.go:356] "Generic (PLEG): container finished" podID="f74c7e5b-34f0-480e-8f2a-63b406788ed5" containerID="bfddf001ae2a0912506c402aa103a328a2d45336e12174bf41a9ec549b0cdc10" exitCode=0
Oct 30 21:46:25.634263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:25.633874    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4" event={"ID":"f74c7e5b-34f0-480e-8f2a-63b406788ed5","Type":"ContainerDied","Data":"bfddf001ae2a0912506c402aa103a328a2d45336e12174bf41a9ec549b0cdc10"}
Oct 30 21:46:25.635928 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:25.635904    4918 generic.go:356] "Generic (PLEG): container finished" podID="08cf8023-9260-4e4a-9aa3-c801e7a6f85b" containerID="a95124a4188ba95cf4aab23f1234fa58d53180d5cfc00a56b13cb652ba6fe7b6" exitCode=0
Oct 30 21:46:25.636053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:25.635982    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7" event={"ID":"08cf8023-9260-4e4a-9aa3-c801e7a6f85b","Type":"ContainerDied","Data":"a95124a4188ba95cf4aab23f1234fa58d53180d5cfc00a56b13cb652ba6fe7b6"}
Oct 30 21:46:25.825747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:25.825717    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:25.961124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:25.960519    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74c7e5b-34f0-480e-8f2a-63b406788ed5-kube-api-access-lk72c\" (UniqueName: \"kubernetes.io/projected/f74c7e5b-34f0-480e-8f2a-63b406788ed5-kube-api-access-lk72c\") pod \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\" (UID: \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\") "
Oct 30 21:46:25.961124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:25.960601    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-openshift-ca\") pod \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\" (UID: \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\") "
Oct 30 21:46:25.961124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:25.960724    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-xks-ca\") pod \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\" (UID: \"f74c7e5b-34f0-480e-8f2a-63b406788ed5\") "
Oct 30 21:46:26.022419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.017029    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-openshift-ca" pod "f74c7e5b-34f0-480e-8f2a-63b406788ed5" (UID: "f74c7e5b-34f0-480e-8f2a-63b406788ed5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:26.037350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.036792    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-xks-ca" pod "f74c7e5b-34f0-480e-8f2a-63b406788ed5" (UID: "f74c7e5b-34f0-480e-8f2a-63b406788ed5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:26.038942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.038899    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74c7e5b-34f0-480e-8f2a-63b406788ed5-kube-api-access-lk72c" pod "f74c7e5b-34f0-480e-8f2a-63b406788ed5" (UID: "f74c7e5b-34f0-480e-8f2a-63b406788ed5"). InnerVolumeSpecName "kube-api-access-lk72c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:26.047643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.047611    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"]
Oct 30 21:46:26.054384 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.054361    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7354c004-msdhr"]
Oct 30 21:46:26.063428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.063382    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk72c\" (UniqueName: \"kubernetes.io/projected/f74c7e5b-34f0-480e-8f2a-63b406788ed5-kube-api-access-lk72c\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:26.063540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.063428    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:26.063540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.063445    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74c7e5b-34f0-480e-8f2a-63b406788ed5-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:26.650398 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.650372    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"
Oct 30 21:46:26.650826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.650414    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4" event={"ID":"f74c7e5b-34f0-480e-8f2a-63b406788ed5","Type":"ContainerDied","Data":"eef7f2be2840b8440e2fdfa7707b010cbc353626d899e768fdf31cb2a179fc67"}
Oct 30 21:46:26.650826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.650460    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eef7f2be2840b8440e2fdfa7707b010cbc353626d899e768fdf31cb2a179fc67"
Oct 30 21:46:26.823897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.823874    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:26.886412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.876315    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-xks-ca\") pod \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\" (UID: \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\") "
Oct 30 21:46:26.913511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.908728    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-kube-api-access-lqdmt\" (UniqueName: \"kubernetes.io/projected/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-kube-api-access-lqdmt\") pod \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\" (UID: \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\") "
Oct 30 21:46:26.913511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.909165    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-openshift-ca\") pod \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\" (UID: \"08cf8023-9260-4e4a-9aa3-c801e7a6f85b\") "
Oct 30 21:46:26.915441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.915279    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-openshift-ca" pod "08cf8023-9260-4e4a-9aa3-c801e7a6f85b" (UID: "08cf8023-9260-4e4a-9aa3-c801e7a6f85b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:26.934418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.926983    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-kube-api-access-lqdmt" pod "08cf8023-9260-4e4a-9aa3-c801e7a6f85b" (UID: "08cf8023-9260-4e4a-9aa3-c801e7a6f85b"). InnerVolumeSpecName "kube-api-access-lqdmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:26.955015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:26.949372    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-xks-ca" pod "08cf8023-9260-4e4a-9aa3-c801e7a6f85b" (UID: "08cf8023-9260-4e4a-9aa3-c801e7a6f85b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:27.016814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:27.016782    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:27.016814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:27.016813    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:27.017014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:27.016830    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqdmt\" (UniqueName: \"kubernetes.io/projected/08cf8023-9260-4e4a-9aa3-c801e7a6f85b-kube-api-access-lqdmt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:27.084549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:27.084513    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fd993b1-2129-44db-9269-2943acd440b7" path="/var/lib/kubelet/pods/3fd993b1-2129-44db-9269-2943acd440b7/volumes"
Oct 30 21:46:27.660485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:27.660457    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"
Oct 30 21:46:27.660874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:27.660518    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7" event={"ID":"08cf8023-9260-4e4a-9aa3-c801e7a6f85b","Type":"ContainerDied","Data":"8132fafc0c1186b35147a2161446d1fb032b83eba3ae3285c46278f75c7b81de"}
Oct 30 21:46:27.660874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:27.660597    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8132fafc0c1186b35147a2161446d1fb032b83eba3ae3285c46278f75c7b81de"
Oct 30 21:46:29.184816 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:46:29.184716    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:46:29.567604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:29.567577    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"
Oct 30 21:46:29.567798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:29.567646    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"
Oct 30 21:46:30.051485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.051449    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"]
Oct 30 21:46:30.057419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.053109    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f74c7e5b-34f0-480e-8f2a-63b406788ed5" containerName="revoke-keys"
Oct 30 21:46:30.057737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.057714    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74c7e5b-34f0-480e-8f2a-63b406788ed5" containerName="revoke-keys"
Oct 30 21:46:30.068415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.058209    4918 cpu_manager.go:414] "containerMap: removing container" podUID="08cf8023-9260-4e4a-9aa3-c801e7a6f85b" containerName="revoke-keys"
Oct 30 21:46:30.068647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.068625    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08cf8023-9260-4e4a-9aa3-c801e7a6f85b" containerName="revoke-keys"
Oct 30 21:46:30.069105 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.069084    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="08cf8023-9260-4e4a-9aa3-c801e7a6f85b" containerName="revoke-keys"
Oct 30 21:46:30.069276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.069261    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74c7e5b-34f0-480e-8f2a-63b406788ed5" containerName="revoke-keys"
Oct 30 21:46:30.113417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.112854    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.139383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.139183    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"payload-processing-external-model-e2e--74dc6735-dockercfg-bqpjt\""
Oct 30 21:46:30.162597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.144029    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"openshift-service-ca.crt\""
Oct 30 21:46:30.162824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.144905    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"routing-overlay\""
Oct 30 21:46:30.162936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.144966    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8072a0\""
Oct 30 21:46:30.163038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.145043    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"kube-root-ca.crt\""
Oct 30 21:46:30.163153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.145096    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8072a0\"/\"agc-cred-32727417a00a\""
Oct 30 21:46:30.184515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.183523    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qxw6\" (UniqueName: \"kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-kube-api-access-4qxw6\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.185331 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.185290    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.186020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.185998    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.186324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.186305    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.189451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.189429    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"]
Oct 30 21:46:30.288298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.288269    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.288516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.288415    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.288516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.288480    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qxw6\" (UniqueName: \"kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-kube-api-access-4qxw6\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.288915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.288874    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.289232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.289208    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.289558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.289535    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.291103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.291084    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.298622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.298601    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qxw6\" (UniqueName: \"kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-kube-api-access-4qxw6\") pod \"payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") " pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.465318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.465236    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:30.921413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.920068    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"]
Oct 30 21:46:30.937703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.937671    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:30.965009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.964982    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:46:30.965134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.965074    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:46:30.982120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.982099    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:46:30.987034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:30.986999    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"]
Oct 30 21:46:31.000239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.000206    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tm2m\" (UniqueName: \"kubernetes.io/projected/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-kube-api-access-7tm2m\") pod \"maas-api-revoke-sub-92eb4bfd-p7nfr\" (UID: \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:31.000550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.000528    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92eb4bfd-p7nfr\" (UID: \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:31.000987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.000965    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92eb4bfd-p7nfr\" (UID: \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:31.019754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.019728    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:46:31.023155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.023131    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:46:31.043113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.043084    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:46:31.102149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.102116    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tm2m\" (UniqueName: \"kubernetes.io/projected/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-kube-api-access-7tm2m\") pod \"maas-api-revoke-sub-92eb4bfd-p7nfr\" (UID: \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:31.102357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.102204    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92eb4bfd-p7nfr\" (UID: \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:31.102495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.102458    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92eb4bfd-p7nfr\" (UID: \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:31.102940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.102915    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92eb4bfd-p7nfr\" (UID: \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:31.104950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.104931    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92eb4bfd-p7nfr\" (UID: \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:31.110695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.110672    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tm2m\" (UniqueName: \"kubernetes.io/projected/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-kube-api-access-7tm2m\") pod \"maas-api-revoke-sub-92eb4bfd-p7nfr\" (UID: \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:31.242220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.242184    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"]
Oct 30 21:46:31.244363 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:46:31.244325    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9425435_5a6a_47f8_8c7f_551e1bc2ab29.slice/crio-1ab5ed16f96584b8caa682687dbfbe41b25be1108c6ff891549e4b2e8fcfddc7 WatchSource:0}: Error finding container 1ab5ed16f96584b8caa682687dbfbe41b25be1108c6ff891549e4b2e8fcfddc7: Status 404 returned error can't find the container with id 1ab5ed16f96584b8caa682687dbfbe41b25be1108c6ff891549e4b2e8fcfddc7
Oct 30 21:46:31.285418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.285365    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:31.581666 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:46:31.581633    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf8f8a1_f957_4f2c_9b3e_0fe68f0e9b59.slice/crio-d7eceb478be0a912148ccef80d513c71a262c3d6b02398d141294e39959923a0 WatchSource:0}: Error finding container d7eceb478be0a912148ccef80d513c71a262c3d6b02398d141294e39959923a0: Status 404 returned error can't find the container with id d7eceb478be0a912148ccef80d513c71a262c3d6b02398d141294e39959923a0
Oct 30 21:46:31.582300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:31.582194    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"]
Oct 30 21:46:32.044046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:32.043525    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr" event={"ID":"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59","Type":"ContainerStarted","Data":"d7eceb478be0a912148ccef80d513c71a262c3d6b02398d141294e39959923a0"}
Oct 30 21:46:32.045705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:32.045682    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr" event={"ID":"a9425435-5a6a-47f8-8c7f-551e1bc2ab29","Type":"ContainerStarted","Data":"3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b"}
Oct 30 21:46:32.045886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:32.045873    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr" event={"ID":"a9425435-5a6a-47f8-8c7f-551e1bc2ab29","Type":"ContainerStarted","Data":"1ab5ed16f96584b8caa682687dbfbe41b25be1108c6ff891549e4b2e8fcfddc7"}
Oct 30 21:46:32.045970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:32.045962    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:32.070884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:32.070839    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr" podStartSLOduration=3.07082375 podStartE2EDuration="3.07082375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:32.066661787 +0000 UTC m=+2705.113220953" watchObservedRunningTime="2026-10-30 21:46:32.07082375 +0000 UTC m=+2705.117382901"
Oct 30 21:46:33.057714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:33.057673    4918 generic.go:356] "Generic (PLEG): container finished" podID="fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59" containerID="515e52bf4982eed9b4cdd012248ff8ddb62544b20003d2290cac7f8988aa7026" exitCode=0
Oct 30 21:46:33.058252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:33.057718    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr" event={"ID":"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59","Type":"ContainerDied","Data":"515e52bf4982eed9b4cdd012248ff8ddb62544b20003d2290cac7f8988aa7026"}
Oct 30 21:46:33.964493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:33.964463    4918 scope.go:122] "RemoveContainer" containerID="6375f2777bf6f1f83db960680d06ef63e4236281726338cefe7deb4e9a133455"
Oct 30 21:46:34.209213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.209178    4918 scope.go:122] "RemoveContainer" containerID="3455d3e8446247d3ab9cf337bbe4fac349962f5e3ab84cd30636a95c1eb10cd3"
Oct 30 21:46:34.319424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.312636    4918 scope.go:122] "RemoveContainer" containerID="2af7c8a6bad9f1a87862179fc773e3de8787aec7ad319d5e6617349c0e23afe7"
Oct 30 21:46:34.340665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.340633    4918 scope.go:122] "RemoveContainer" containerID="4cd16d7a100c885a20154070e25002b6abe464ed6ee9c5553f1c2b52cb65b0d4"
Oct 30 21:46:34.861120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.861096    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:34.892065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.892033    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-xks-ca\") pod \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\" (UID: \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\") "
Oct 30 21:46:34.892258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.892235    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-openshift-ca\") pod \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\" (UID: \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\") "
Oct 30 21:46:34.892321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.892280    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-kube-api-access-7tm2m\" (UniqueName: \"kubernetes.io/projected/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-kube-api-access-7tm2m\") pod \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\" (UID: \"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59\") "
Oct 30 21:46:34.892863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.892820    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-openshift-ca" pod "fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59" (UID: "fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:34.893130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.893108    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:34.895841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.895812    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-xks-ca" pod "fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59" (UID: "fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:34.896480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.896456    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-kube-api-access-7tm2m" pod "fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59" (UID: "fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59"). InnerVolumeSpecName "kube-api-access-7tm2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:34.994782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.994749    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tm2m\" (UniqueName: \"kubernetes.io/projected/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-kube-api-access-7tm2m\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:34.994782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:34.994775    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:35.247542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:35.247512    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"
Oct 30 21:46:35.247542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:35.247540    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr" event={"ID":"fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59","Type":"ContainerDied","Data":"d7eceb478be0a912148ccef80d513c71a262c3d6b02398d141294e39959923a0"}
Oct 30 21:46:35.248075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:35.247588    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7eceb478be0a912148ccef80d513c71a262c3d6b02398d141294e39959923a0"
Oct 30 21:46:35.637287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:35.637256    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"]
Oct 30 21:46:36.461033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.460993    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"]
Oct 30 21:46:36.463100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.463072    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59" containerName="revoke-keys"
Oct 30 21:46:36.463100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.463097    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59" containerName="revoke-keys"
Oct 30 21:46:36.463341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.463326    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59" containerName="revoke-keys"
Oct 30 21:46:36.468925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.468901    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.499626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.499596    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"]
Oct 30 21:46:36.519865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.519813    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.520065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.520047    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.520144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.520115    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.520300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.520278    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.520383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.520318    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.520515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.520440    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvcdx\" (UniqueName: \"kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-kube-api-access-xvcdx\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.520515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.520507    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.520619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.520597    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.520656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.520646    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.526573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.526383    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-cf51d4-openshift-default-dockercfg-jq2sn\""
Oct 30 21:46:36.623000 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.622964    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623095    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623130    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623213    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623234    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623278    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvcdx\" (UniqueName: \"kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-kube-api-access-xvcdx\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623321    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623364    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623445    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623624    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-data\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623690    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-credential-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623779    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istiod-ca-cert\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623889 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623867    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-certs\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.623981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.623961    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-socket\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.625678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.625657    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-envoy\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.625902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.625885    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-podinfo\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.644250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.644198    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvcdx\" (UniqueName: \"kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-kube-api-access-xvcdx\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.645679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.645264    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-token\") pod \"e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") " pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:36.793415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:36.792764    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:37.370327 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:46:37.370294    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5cf8324_5cf6_49d2_8e8d_2ec76605e96c.slice/crio-604441b3ab027a9b88a5572e7a0e2ab2db3863012f6306bd84df2c179158e547 WatchSource:0}: Error finding container 604441b3ab027a9b88a5572e7a0e2ab2db3863012f6306bd84df2c179158e547: Status 404 returned error can't find the container with id 604441b3ab027a9b88a5572e7a0e2ab2db3863012f6306bd84df2c179158e547
Oct 30 21:46:37.372592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:37.372566    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"]
Oct 30 21:46:37.373305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:37.373276    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:46:37.373366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:37.373345    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:46:37.373430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:37.373373    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:46:38.405418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:38.405253    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" event={"ID":"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c","Type":"ContainerStarted","Data":"7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2"}
Oct 30 21:46:38.405418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:38.405318    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" event={"ID":"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c","Type":"ContainerStarted","Data":"604441b3ab027a9b88a5572e7a0e2ab2db3863012f6306bd84df2c179158e547"}
Oct 30 21:46:38.458417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:38.447727    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" podStartSLOduration=2.447707396 podStartE2EDuration="2.447707396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:38.433840646 +0000 UTC m=+2711.480399793" watchObservedRunningTime="2026-10-30 21:46:38.447707396 +0000 UTC m=+2711.494266542"
Oct 30 21:46:38.798413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:38.798059    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:38.820421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:38.820038    4918 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:46:38.820421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:38.820113    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused"
Oct 30 21:46:39.829419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:39.823375    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:40.457599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:40.457549    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:40.458698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:40.458670    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:46:40.637682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:40.637644    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks" podUID="c465785a-a77a-4721-b3a5-551b9ff2e394" containerName="payload-processing" containerID="cri-o://2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee" gracePeriod=25
Oct 30 21:46:41.040901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.040877    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:46:41.103044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.102961    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-plugins-config-volume\") pod \"c465785a-a77a-4721-b3a5-551b9ff2e394\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") "
Oct 30 21:46:41.103290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.103270    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-kube-api-access-smdws\" (UniqueName: \"kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-kube-api-access-smdws\") pod \"c465785a-a77a-4721-b3a5-551b9ff2e394\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") "
Oct 30 21:46:41.103378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.103333    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-routing-overlay-volume\") pod \"c465785a-a77a-4721-b3a5-551b9ff2e394\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") "
Oct 30 21:46:41.103543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.103525    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-provider-credentials\") pod \"c465785a-a77a-4721-b3a5-551b9ff2e394\" (UID: \"c465785a-a77a-4721-b3a5-551b9ff2e394\") "
Oct 30 21:46:41.105058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.105028    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-routing-overlay-volume" pod "c465785a-a77a-4721-b3a5-551b9ff2e394" (UID: "c465785a-a77a-4721-b3a5-551b9ff2e394"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:41.105497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.105470    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-plugins-config-volume" pod "c465785a-a77a-4721-b3a5-551b9ff2e394" (UID: "c465785a-a77a-4721-b3a5-551b9ff2e394"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:41.107632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.107602    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-kube-api-access-smdws" pod "c465785a-a77a-4721-b3a5-551b9ff2e394" (UID: "c465785a-a77a-4721-b3a5-551b9ff2e394"). InnerVolumeSpecName "kube-api-access-smdws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:41.107829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.107799    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-provider-credentials" pod "c465785a-a77a-4721-b3a5-551b9ff2e394" (UID: "c465785a-a77a-4721-b3a5-551b9ff2e394"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:41.108717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.108693    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"]
Oct 30 21:46:41.110019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.109996    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c465785a-a77a-4721-b3a5-551b9ff2e394" containerName="payload-processing"
Oct 30 21:46:41.110123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.110019    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c465785a-a77a-4721-b3a5-551b9ff2e394" containerName="payload-processing"
Oct 30 21:46:41.110317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.110303    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c465785a-a77a-4721-b3a5-551b9ff2e394" containerName="payload-processing"
Oct 30 21:46:41.121050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.121020    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:41.126446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.126420    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"]
Oct 30 21:46:41.127766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.127183    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:46:41.127766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.127221    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:46:41.127766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.127358    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:46:41.127766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.127511    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:46:41.219444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.215867    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb76d7c-6qv9q\" (UID: \"a0f75d56-ce02-465b-8746-ab34b97bfd66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:41.219444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.216109    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb76d7c-6qv9q\" (UID: \"a0f75d56-ce02-465b-8746-ab34b97bfd66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:41.219444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.216371    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvjx\" (UniqueName: \"kubernetes.io/projected/a0f75d56-ce02-465b-8746-ab34b97bfd66-kube-api-access-7zvjx\") pod \"maas-api-revoke-sub-cdb76d7c-6qv9q\" (UID: \"a0f75d56-ce02-465b-8746-ab34b97bfd66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:41.219444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.216510    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:41.219444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.216532    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smdws\" (UniqueName: \"kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-kube-api-access-smdws\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:41.219444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.216546    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c465785a-a77a-4721-b3a5-551b9ff2e394-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:41.219444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.216560    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c465785a-a77a-4721-b3a5-551b9ff2e394-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:41.317823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.317777    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb76d7c-6qv9q\" (UID: \"a0f75d56-ce02-465b-8746-ab34b97bfd66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:41.318010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.317945    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvjx\" (UniqueName: \"kubernetes.io/projected/a0f75d56-ce02-465b-8746-ab34b97bfd66-kube-api-access-7zvjx\") pod \"maas-api-revoke-sub-cdb76d7c-6qv9q\" (UID: \"a0f75d56-ce02-465b-8746-ab34b97bfd66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:41.318079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.318057    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb76d7c-6qv9q\" (UID: \"a0f75d56-ce02-465b-8746-ab34b97bfd66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:41.318707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.318683    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdb76d7c-6qv9q\" (UID: \"a0f75d56-ce02-465b-8746-ab34b97bfd66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:41.320512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.320489    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdb76d7c-6qv9q\" (UID: \"a0f75d56-ce02-465b-8746-ab34b97bfd66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:41.328093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.328064    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvjx\" (UniqueName: \"kubernetes.io/projected/a0f75d56-ce02-465b-8746-ab34b97bfd66-kube-api-access-7zvjx\") pod \"maas-api-revoke-sub-cdb76d7c-6qv9q\" (UID: \"a0f75d56-ce02-465b-8746-ab34b97bfd66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:41.469463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.464294    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:41.555923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.555369    4918 generic.go:356] "Generic (PLEG): container finished" podID="c465785a-a77a-4721-b3a5-551b9ff2e394" containerID="2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee" exitCode=0
Oct 30 21:46:41.556887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.556760    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"
Oct 30 21:46:41.567591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.566547    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks" event={"ID":"c465785a-a77a-4721-b3a5-551b9ff2e394","Type":"ContainerDied","Data":"2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee"}
Oct 30 21:46:41.567591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.566629    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks" event={"ID":"c465785a-a77a-4721-b3a5-551b9ff2e394","Type":"ContainerDied","Data":"62a1a023a41cbe8f6e150628d8a49bc7a34a5757854f7d1c572be86da5bcbd2f"}
Oct 30 21:46:41.567591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.566660    4918 scope.go:122] "RemoveContainer" containerID="2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee"
Oct 30 21:46:41.586540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.586109    4918 scope.go:122] "RemoveContainer" containerID="2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee"
Oct 30 21:46:41.586540 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:46:41.586508    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee\": container with ID starting with 2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee not found: ID does not exist" containerID="2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee"
Oct 30 21:46:41.586705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.586541    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee"} err="failed to get container status \"2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee\": rpc error: code = NotFound desc = could not find container \"2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee\": container with ID starting with 2ccbe9c5b27bbd8d8c725fbb8b4423d6bdfe7bf810f688f80072047b139ad1ee not found: ID does not exist"
Oct 30 21:46:41.604083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.604054    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"]
Oct 30 21:46:41.609518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.609494    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d847fc69b-fpqks"]
Oct 30 21:46:41.853364 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:46:41.853328    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f75d56_ce02_465b_8746_ab34b97bfd66.slice/crio-0443a0d1d00e2de19d43c29173ce019a7f24f6e70781ea245e58f27c466ff4d8 WatchSource:0}: Error finding container 0443a0d1d00e2de19d43c29173ce019a7f24f6e70781ea245e58f27c466ff4d8: Status 404 returned error can't find the container with id 0443a0d1d00e2de19d43c29173ce019a7f24f6e70781ea245e58f27c466ff4d8
Oct 30 21:46:41.855012 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:41.854986    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"]
Oct 30 21:46:42.263127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.263096    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"]
Oct 30 21:46:42.268352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.268333    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"
Oct 30 21:46:42.277407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.277369    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"]
Oct 30 21:46:42.279714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.279693    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-cf51d4-dockercfg-8nqjc\""
Oct 30 21:46:42.279964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.279929    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-cf51d4\""
Oct 30 21:46:42.299495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.299461    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"]
Oct 30 21:46:42.308382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.306720    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"
Oct 30 21:46:42.314276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.314246    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"]
Oct 30 21:46:42.382420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.382150    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7000f0-3e79-4852-9992-b07ec806326c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr\" (UID: \"ff7000f0-3e79-4852-9992-b07ec806326c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"
Oct 30 21:46:42.382420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.382347    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fs9n\" (UniqueName: \"kubernetes.io/projected/ff7000f0-3e79-4852-9992-b07ec806326c-kube-api-access-4fs9n\") pod \"payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr\" (UID: \"ff7000f0-3e79-4852-9992-b07ec806326c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"
Oct 30 21:46:42.382744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.382490    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94csr\" (UniqueName: \"kubernetes.io/projected/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-kube-api-access-94csr\") pod \"payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g\" (UID: \"c7167d31-70cc-4ed1-9f13-9fea12fb1a85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"
Oct 30 21:46:42.382744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.382557    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g\" (UID: \"c7167d31-70cc-4ed1-9f13-9fea12fb1a85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"
Oct 30 21:46:42.545443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.544817    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7000f0-3e79-4852-9992-b07ec806326c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr\" (UID: \"ff7000f0-3e79-4852-9992-b07ec806326c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"
Oct 30 21:46:42.545443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.544949    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fs9n\" (UniqueName: \"kubernetes.io/projected/ff7000f0-3e79-4852-9992-b07ec806326c-kube-api-access-4fs9n\") pod \"payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr\" (UID: \"ff7000f0-3e79-4852-9992-b07ec806326c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"
Oct 30 21:46:42.545443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.545035    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94csr\" (UniqueName: \"kubernetes.io/projected/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-kube-api-access-94csr\") pod \"payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g\" (UID: \"c7167d31-70cc-4ed1-9f13-9fea12fb1a85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"
Oct 30 21:46:42.545443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.545083    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g\" (UID: \"c7167d31-70cc-4ed1-9f13-9fea12fb1a85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"
Oct 30 21:46:42.580510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.546046    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g\" (UID: \"c7167d31-70cc-4ed1-9f13-9fea12fb1a85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"
Oct 30 21:46:42.734900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.733829    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94csr\" (UniqueName: \"kubernetes.io/projected/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-kube-api-access-94csr\") pod \"payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g\" (UID: \"c7167d31-70cc-4ed1-9f13-9fea12fb1a85\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"
Oct 30 21:46:42.734900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.734764    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"
Oct 30 21:46:42.794252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.794084    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fs9n\" (UniqueName: \"kubernetes.io/projected/ff7000f0-3e79-4852-9992-b07ec806326c-kube-api-access-4fs9n\") pod \"payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr\" (UID: \"ff7000f0-3e79-4852-9992-b07ec806326c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"
Oct 30 21:46:42.802460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.801336    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q" event={"ID":"a0f75d56-ce02-465b-8746-ab34b97bfd66","Type":"ContainerStarted","Data":"0443a0d1d00e2de19d43c29173ce019a7f24f6e70781ea245e58f27c466ff4d8"}
Oct 30 21:46:42.804718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.804687    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7000f0-3e79-4852-9992-b07ec806326c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr\" (UID: \"ff7000f0-3e79-4852-9992-b07ec806326c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"
Oct 30 21:46:42.891435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:42.887726    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"
Oct 30 21:46:43.061436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.061367    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:43.091315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.091272    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c465785a-a77a-4721-b3a5-551b9ff2e394" path="/var/lib/kubelet/pods/c465785a-a77a-4721-b3a5-551b9ff2e394/volumes"
Oct 30 21:46:43.332549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.327440    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"]
Oct 30 21:46:43.343975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.343934    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"]
Oct 30 21:46:43.816747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.816714    4918 generic.go:356] "Generic (PLEG): container finished" podID="a0f75d56-ce02-465b-8746-ab34b97bfd66" containerID="016de08f7e249e77fc681b6d5ad8b845786522e51e923a75e786c45fbf0ab977" exitCode=0
Oct 30 21:46:43.816919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.816795    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q" event={"ID":"a0f75d56-ce02-465b-8746-ab34b97bfd66","Type":"ContainerDied","Data":"016de08f7e249e77fc681b6d5ad8b845786522e51e923a75e786c45fbf0ab977"}
Oct 30 21:46:43.818489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.818463    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g" event={"ID":"c7167d31-70cc-4ed1-9f13-9fea12fb1a85","Type":"ContainerStarted","Data":"4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a"}
Oct 30 21:46:43.818622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.818508    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g" event={"ID":"c7167d31-70cc-4ed1-9f13-9fea12fb1a85","Type":"ContainerStarted","Data":"fc303a371df505c4108f55ec15032a2a0ba961a309459afbd491d35e4ca36b3a"}
Oct 30 21:46:43.818622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.818536    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"
Oct 30 21:46:43.820321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.820296    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr" event={"ID":"ff7000f0-3e79-4852-9992-b07ec806326c","Type":"ContainerStarted","Data":"2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55"}
Oct 30 21:46:43.820439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.820336    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr" event={"ID":"ff7000f0-3e79-4852-9992-b07ec806326c","Type":"ContainerStarted","Data":"dc64a8eb431bbb714b32e523e0b49da3ccc7e89571c55dd7eb2cd54e7ff8e7d4"}
Oct 30 21:46:43.820439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.820370    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"
Oct 30 21:46:43.885559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.885508    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr" podStartSLOduration=1.8854955279999999 podStartE2EDuration="1.885495528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:43.883529024 +0000 UTC m=+2716.930088208" watchObservedRunningTime="2026-10-30 21:46:43.885495528 +0000 UTC m=+2716.932054705"
Oct 30 21:46:43.913344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:43.913295    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g" podStartSLOduration=1.913283399 podStartE2EDuration="1.913283399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:43.907499083 +0000 UTC m=+2716.954058261" watchObservedRunningTime="2026-10-30 21:46:43.913283399 +0000 UTC m=+2716.959842575"
Oct 30 21:46:45.065823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.065789    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:45.182471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.169500    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-xks-ca\") pod \"a0f75d56-ce02-465b-8746-ab34b97bfd66\" (UID: \"a0f75d56-ce02-465b-8746-ab34b97bfd66\") "
Oct 30 21:46:45.182471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.169593    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-openshift-ca\") pod \"a0f75d56-ce02-465b-8746-ab34b97bfd66\" (UID: \"a0f75d56-ce02-465b-8746-ab34b97bfd66\") "
Oct 30 21:46:45.182471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.169946    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f75d56-ce02-465b-8746-ab34b97bfd66-kube-api-access-7zvjx\" (UniqueName: \"kubernetes.io/projected/a0f75d56-ce02-465b-8746-ab34b97bfd66-kube-api-access-7zvjx\") pod \"a0f75d56-ce02-465b-8746-ab34b97bfd66\" (UID: \"a0f75d56-ce02-465b-8746-ab34b97bfd66\") "
Oct 30 21:46:45.214417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.204690    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-openshift-ca" pod "a0f75d56-ce02-465b-8746-ab34b97bfd66" (UID: "a0f75d56-ce02-465b-8746-ab34b97bfd66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:45.230418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.222816    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:45.251551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.251513    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-xks-ca" pod "a0f75d56-ce02-465b-8746-ab34b97bfd66" (UID: "a0f75d56-ce02-465b-8746-ab34b97bfd66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:45.268924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.264746    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f75d56-ce02-465b-8746-ab34b97bfd66-kube-api-access-7zvjx" pod "a0f75d56-ce02-465b-8746-ab34b97bfd66" (UID: "a0f75d56-ce02-465b-8746-ab34b97bfd66"). InnerVolumeSpecName "kube-api-access-7zvjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:45.324796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.324757    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zvjx\" (UniqueName: \"kubernetes.io/projected/a0f75d56-ce02-465b-8746-ab34b97bfd66-kube-api-access-7zvjx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:45.324796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.324799    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f75d56-ce02-465b-8746-ab34b97bfd66-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:45.921764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.921729    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q" event={"ID":"a0f75d56-ce02-465b-8746-ab34b97bfd66","Type":"ContainerDied","Data":"0443a0d1d00e2de19d43c29173ce019a7f24f6e70781ea245e58f27c466ff4d8"}
Oct 30 21:46:45.922041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.922026    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0443a0d1d00e2de19d43c29173ce019a7f24f6e70781ea245e58f27c466ff4d8"
Oct 30 21:46:45.922195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:45.922185    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"
Oct 30 21:46:52.643826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:52.643792    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"]
Oct 30 21:46:53.322413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.321419    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"]
Oct 30 21:46:53.335465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.322870    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a0f75d56-ce02-465b-8746-ab34b97bfd66" containerName="revoke-keys"
Oct 30 21:46:53.335465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.322896    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f75d56-ce02-465b-8746-ab34b97bfd66" containerName="revoke-keys"
Oct 30 21:46:53.335465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.329602    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f75d56-ce02-465b-8746-ab34b97bfd66" containerName="revoke-keys"
Oct 30 21:46:53.349703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.348684    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:53.363570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.363540    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:46:53.363798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.363752    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:46:53.363916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.363840    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:46:53.373774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.373729    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6qmp\" (UniqueName: \"kubernetes.io/projected/661569f6-7608-406c-8d65-920ceb80ff22-kube-api-access-m6qmp\") pod \"maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh\" (UID: \"661569f6-7608-406c-8d65-920ceb80ff22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:53.374145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.374124    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh\" (UID: \"661569f6-7608-406c-8d65-920ceb80ff22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:53.374514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.374493    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh\" (UID: \"661569f6-7608-406c-8d65-920ceb80ff22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:53.402411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.399445    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"]
Oct 30 21:46:53.418432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.413902    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:46:53.476984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.476952    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh\" (UID: \"661569f6-7608-406c-8d65-920ceb80ff22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:53.477286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.477158    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6qmp\" (UniqueName: \"kubernetes.io/projected/661569f6-7608-406c-8d65-920ceb80ff22-kube-api-access-m6qmp\") pod \"maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh\" (UID: \"661569f6-7608-406c-8d65-920ceb80ff22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:53.477286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.477253    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh\" (UID: \"661569f6-7608-406c-8d65-920ceb80ff22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:53.477950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.477907    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh\" (UID: \"661569f6-7608-406c-8d65-920ceb80ff22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:53.480552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.480526    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh\" (UID: \"661569f6-7608-406c-8d65-920ceb80ff22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:53.488281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.488237    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6qmp\" (UniqueName: \"kubernetes.io/projected/661569f6-7608-406c-8d65-920ceb80ff22-kube-api-access-m6qmp\") pod \"maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh\" (UID: \"661569f6-7608-406c-8d65-920ceb80ff22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:53.718467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:53.717696    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:54.352735 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:46:54.352704    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661569f6_7608_406c_8d65_920ceb80ff22.slice/crio-03e346986329f2d044dd151936c2064b4864b36c5025d0bb8406b138a29d4b84 WatchSource:0}: Error finding container 03e346986329f2d044dd151936c2064b4864b36c5025d0bb8406b138a29d4b84: Status 404 returned error can't find the container with id 03e346986329f2d044dd151936c2064b4864b36c5025d0bb8406b138a29d4b84
Oct 30 21:46:54.354002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:54.353978    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"]
Oct 30 21:46:54.847480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:54.847448    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"
Oct 30 21:46:54.847946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:54.847739    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"
Oct 30 21:46:55.360848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:55.360816    4918 generic.go:356] "Generic (PLEG): container finished" podID="661569f6-7608-406c-8d65-920ceb80ff22" containerID="b9f0685d6563057005a71319d1fafed00a9aa20198e94c52e3c847f04846f14a" exitCode=0
Oct 30 21:46:55.361130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:55.360908    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh" event={"ID":"661569f6-7608-406c-8d65-920ceb80ff22","Type":"ContainerDied","Data":"b9f0685d6563057005a71319d1fafed00a9aa20198e94c52e3c847f04846f14a"}
Oct 30 21:46:55.361130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:55.360958    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh" event={"ID":"661569f6-7608-406c-8d65-920ceb80ff22","Type":"ContainerStarted","Data":"03e346986329f2d044dd151936c2064b4864b36c5025d0bb8406b138a29d4b84"}
Oct 30 21:46:56.650228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:56.650202    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:56.683249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:56.683222    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-openshift-ca\") pod \"661569f6-7608-406c-8d65-920ceb80ff22\" (UID: \"661569f6-7608-406c-8d65-920ceb80ff22\") "
Oct 30 21:46:56.683442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:56.683430    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661569f6-7608-406c-8d65-920ceb80ff22-kube-api-access-m6qmp\" (UniqueName: \"kubernetes.io/projected/661569f6-7608-406c-8d65-920ceb80ff22-kube-api-access-m6qmp\") pod \"661569f6-7608-406c-8d65-920ceb80ff22\" (UID: \"661569f6-7608-406c-8d65-920ceb80ff22\") "
Oct 30 21:46:56.683502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:56.683457    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-xks-ca\") pod \"661569f6-7608-406c-8d65-920ceb80ff22\" (UID: \"661569f6-7608-406c-8d65-920ceb80ff22\") "
Oct 30 21:46:56.683673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:56.683644    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-openshift-ca" pod "661569f6-7608-406c-8d65-920ceb80ff22" (UID: "661569f6-7608-406c-8d65-920ceb80ff22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:56.684138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:56.684083    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:56.685835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:56.685800    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-xks-ca" pod "661569f6-7608-406c-8d65-920ceb80ff22" (UID: "661569f6-7608-406c-8d65-920ceb80ff22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:56.686440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:56.686417    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661569f6-7608-406c-8d65-920ceb80ff22-kube-api-access-m6qmp" pod "661569f6-7608-406c-8d65-920ceb80ff22" (UID: "661569f6-7608-406c-8d65-920ceb80ff22"). InnerVolumeSpecName "kube-api-access-m6qmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:56.785731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:56.785694    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6qmp\" (UniqueName: \"kubernetes.io/projected/661569f6-7608-406c-8d65-920ceb80ff22-kube-api-access-m6qmp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:56.785731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:56.785725    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/661569f6-7608-406c-8d65-920ceb80ff22-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:57.392997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:57.392905    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"
Oct 30 21:46:57.392997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:57.392940    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh" event={"ID":"661569f6-7608-406c-8d65-920ceb80ff22","Type":"ContainerDied","Data":"03e346986329f2d044dd151936c2064b4864b36c5025d0bb8406b138a29d4b84"}
Oct 30 21:46:57.392997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:57.392989    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03e346986329f2d044dd151936c2064b4864b36c5025d0bb8406b138a29d4b84"
Oct 30 21:46:57.644429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:57.644336    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr" podUID="a9425435-5a6a-47f8-8c7f-551e1bc2ab29" containerName="payload-processing" containerID="cri-o://3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b" gracePeriod=25
Oct 30 21:46:57.948325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:57.948296    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:58.006863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.006824    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-routing-overlay-volume\") pod \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") "
Oct 30 21:46:58.007031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.007014    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-plugins-config-volume\") pod \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") "
Oct 30 21:46:58.007160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.007139    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-routing-overlay-volume" pod "a9425435-5a6a-47f8-8c7f-551e1bc2ab29" (UID: "a9425435-5a6a-47f8-8c7f-551e1bc2ab29"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:58.007286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.007268    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-provider-credentials\") pod \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") "
Oct 30 21:46:58.007350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.007334    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-kube-api-access-4qxw6\" (UniqueName: \"kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-kube-api-access-4qxw6\") pod \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\" (UID: \"a9425435-5a6a-47f8-8c7f-551e1bc2ab29\") "
Oct 30 21:46:58.007556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.007531    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-plugins-config-volume" pod "a9425435-5a6a-47f8-8c7f-551e1bc2ab29" (UID: "a9425435-5a6a-47f8-8c7f-551e1bc2ab29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:46:58.008186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.008157    4918 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-routing-overlay-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:58.008315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.008189    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:58.009681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.009654    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-kube-api-access-4qxw6" pod "a9425435-5a6a-47f8-8c7f-551e1bc2ab29" (UID: "a9425435-5a6a-47f8-8c7f-551e1bc2ab29"). InnerVolumeSpecName "kube-api-access-4qxw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:58.009757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.009689    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-provider-credentials" pod "a9425435-5a6a-47f8-8c7f-551e1bc2ab29" (UID: "a9425435-5a6a-47f8-8c7f-551e1bc2ab29"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:58.110430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.110354    4918 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-provider-credentials\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:58.110430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.110383    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qxw6\" (UniqueName: \"kubernetes.io/projected/a9425435-5a6a-47f8-8c7f-551e1bc2ab29-kube-api-access-4qxw6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:58.404942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.404901    4918 generic.go:356] "Generic (PLEG): container finished" podID="a9425435-5a6a-47f8-8c7f-551e1bc2ab29" containerID="3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b" exitCode=0
Oct 30 21:46:58.405157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.404971    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"
Oct 30 21:46:58.405157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.405059    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr" event={"ID":"a9425435-5a6a-47f8-8c7f-551e1bc2ab29","Type":"ContainerDied","Data":"3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b"}
Oct 30 21:46:58.405157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.405103    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr" event={"ID":"a9425435-5a6a-47f8-8c7f-551e1bc2ab29","Type":"ContainerDied","Data":"1ab5ed16f96584b8caa682687dbfbe41b25be1108c6ff891549e4b2e8fcfddc7"}
Oct 30 21:46:58.405157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.405126    4918 scope.go:122] "RemoveContainer" containerID="3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b"
Oct 30 21:46:58.419261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.419238    4918 scope.go:122] "RemoveContainer" containerID="3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b"
Oct 30 21:46:58.419674 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:46:58.419654    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b\": container with ID starting with 3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b not found: ID does not exist" containerID="3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b"
Oct 30 21:46:58.419768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.419682    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b"} err="failed to get container status \"3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b\": rpc error: code = NotFound desc = could not find container \"3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b\": container with ID starting with 3a32e795cdcbc5767ac349b9a3934182958c4ab43976eb5ae691d0f84311e92b not found: ID does not exist"
Oct 30 21:46:58.438711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.438671    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"]
Oct 30 21:46:58.444092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:58.444053    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8072a0/payload-processing-external-model-e2e-extmod-a-8072a0-66bcwswxr"]
Oct 30 21:46:59.180753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:46:59.180726    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9425435-5a6a-47f8-8c7f-551e1bc2ab29" path="/var/lib/kubelet/pods/a9425435-5a6a-47f8-8c7f-551e1bc2ab29/volumes"
Oct 30 21:46:59.207897 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:46:59.207864    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:47:06.050610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:06.050574    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"]
Oct 30 21:47:06.055588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:06.055563    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-729802-n7bsh"]
Oct 30 21:47:07.134157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:07.133958    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661569f6-7608-406c-8d65-920ceb80ff22" path="/var/lib/kubelet/pods/661569f6-7608-406c-8d65-920ceb80ff22/volumes"
Oct 30 21:47:08.038218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.038175    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"]
Oct 30 21:47:08.041214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.041189    4918 cpu_manager.go:414] "containerMap: removing container" podUID="661569f6-7608-406c-8d65-920ceb80ff22" containerName="revoke-keys"
Oct 30 21:47:08.041381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.041366    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661569f6-7608-406c-8d65-920ceb80ff22" containerName="revoke-keys"
Oct 30 21:47:08.041552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.041536    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a9425435-5a6a-47f8-8c7f-551e1bc2ab29" containerName="payload-processing"
Oct 30 21:47:08.041679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.041662    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9425435-5a6a-47f8-8c7f-551e1bc2ab29" containerName="payload-processing"
Oct 30 21:47:08.042063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.042044    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="661569f6-7608-406c-8d65-920ceb80ff22" containerName="revoke-keys"
Oct 30 21:47:08.042188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.042176    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9425435-5a6a-47f8-8c7f-551e1bc2ab29" containerName="payload-processing"
Oct 30 21:47:08.058550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.058528    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"]
Oct 30 21:47:08.058915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.058834    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"
Oct 30 21:47:08.068690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.068550    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3d9d7b-dockercfg-s6cd7\""
Oct 30 21:47:08.094558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.093146    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3d9d7b\""
Oct 30 21:47:08.228413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.221942    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w22cq\" (UniqueName: \"kubernetes.io/projected/89194be2-23cd-4416-a72e-d0bb5f928e3f-kube-api-access-w22cq\") pod \"payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj\" (UID: \"89194be2-23cd-4416-a72e-d0bb5f928e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"
Oct 30 21:47:08.228413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.222054    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89194be2-23cd-4416-a72e-d0bb5f928e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj\" (UID: \"89194be2-23cd-4416-a72e-d0bb5f928e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"
Oct 30 21:47:08.357465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.355959    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w22cq\" (UniqueName: \"kubernetes.io/projected/89194be2-23cd-4416-a72e-d0bb5f928e3f-kube-api-access-w22cq\") pod \"payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj\" (UID: \"89194be2-23cd-4416-a72e-d0bb5f928e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"
Oct 30 21:47:08.357465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.356062    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89194be2-23cd-4416-a72e-d0bb5f928e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj\" (UID: \"89194be2-23cd-4416-a72e-d0bb5f928e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"
Oct 30 21:47:08.420260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.410966    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89194be2-23cd-4416-a72e-d0bb5f928e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj\" (UID: \"89194be2-23cd-4416-a72e-d0bb5f928e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"
Oct 30 21:47:08.580418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.572216    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w22cq\" (UniqueName: \"kubernetes.io/projected/89194be2-23cd-4416-a72e-d0bb5f928e3f-kube-api-access-w22cq\") pod \"payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj\" (UID: \"89194be2-23cd-4416-a72e-d0bb5f928e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"
Oct 30 21:47:08.707458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:08.704735    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"
Oct 30 21:47:09.600705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:09.600671    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"]
Oct 30 21:47:10.631415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:10.630510    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:10.792426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:10.784612    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj" event={"ID":"89194be2-23cd-4416-a72e-d0bb5f928e3f","Type":"ContainerStarted","Data":"fbb4c46145affdceb23accafc0a84848c6f29dc55300de62a79d9ee52d4708d6"}
Oct 30 21:47:10.829517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:10.814845    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:10.930468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:10.929595    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:10.930468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:10.930087    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:47:10.930468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:10.930420    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": context deadline exceeded"
Oct 30 21:47:10.953413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:10.931691    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:47:10.953413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:10.950099    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:10.972888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:10.954982    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:11.210484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:11.145093    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:47:11.421455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:11.420506    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:11.421455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:11.420506    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:11.798645 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:11.798610    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj" event={"ID":"89194be2-23cd-4416-a72e-d0bb5f928e3f","Type":"ContainerStarted","Data":"ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c"}
Oct 30 21:47:11.799135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:11.798725    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"
Oct 30 21:47:11.820643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:11.820590    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj" podStartSLOduration=4.820576256 podStartE2EDuration="4.820576256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:11.817635109 +0000 UTC m=+2744.864194254" watchObservedRunningTime="2026-10-30 21:47:11.820576256 +0000 UTC m=+2744.867135394"
Oct 30 21:47:14.010090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:14.010043    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"]
Oct 30 21:47:14.010978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:14.010939    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy" containerID="cri-o://4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827" gracePeriod=30
Oct 30 21:47:14.744153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:14.744119    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"]
Oct 30 21:47:14.750377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:14.750356    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:14.813401 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:14.797710    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:47:14.813401 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:14.797859    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:47:14.813401 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:14.797960    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:47:14.813401 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:14.798008    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:47:14.813401 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:14.800444    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"]
Oct 30 21:47:14.958233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:14.958196    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9jvh\" (UniqueName: \"kubernetes.io/projected/1ab33427-b274-4a08-86a6-a876c71057ab-kube-api-access-h9jvh\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5\" (UID: \"1ab33427-b274-4a08-86a6-a876c71057ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:14.958520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:14.958497    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5\" (UID: \"1ab33427-b274-4a08-86a6-a876c71057ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:14.958636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:14.958614    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5\" (UID: \"1ab33427-b274-4a08-86a6-a876c71057ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:15.060347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:15.060264    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9jvh\" (UniqueName: \"kubernetes.io/projected/1ab33427-b274-4a08-86a6-a876c71057ab-kube-api-access-h9jvh\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5\" (UID: \"1ab33427-b274-4a08-86a6-a876c71057ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:15.060910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:15.060527    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5\" (UID: \"1ab33427-b274-4a08-86a6-a876c71057ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:15.060910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:15.060600    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5\" (UID: \"1ab33427-b274-4a08-86a6-a876c71057ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:15.061209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:15.061177    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5\" (UID: \"1ab33427-b274-4a08-86a6-a876c71057ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:15.063095 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:15.063075    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5\" (UID: \"1ab33427-b274-4a08-86a6-a876c71057ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:15.075239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:15.075220    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9jvh\" (UniqueName: \"kubernetes.io/projected/1ab33427-b274-4a08-86a6-a876c71057ab-kube-api-access-h9jvh\") pod \"maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5\" (UID: \"1ab33427-b274-4a08-86a6-a876c71057ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:15.366693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:15.366610    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:15.665497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:15.665459    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"]
Oct 30 21:47:15.984654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:15.984615    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5" event={"ID":"1ab33427-b274-4a08-86a6-a876c71057ab","Type":"ContainerStarted","Data":"01fb6ec58f304c83358a2737f501f77b77a29f72785eaf8bc56c319020ba9e88"}
Oct 30 21:47:16.884263 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:47:16.884232    4918 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab33427_b274_4a08_86a6_a876c71057ab.slice/crio-conmon-107ebd331d6be44a3480c1ab37b276b1e96c8bd7a96ce8a09291d593e319c854.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:47:16.997255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:16.997221    4918 generic.go:356] "Generic (PLEG): container finished" podID="1ab33427-b274-4a08-86a6-a876c71057ab" containerID="107ebd331d6be44a3480c1ab37b276b1e96c8bd7a96ce8a09291d593e319c854" exitCode=0
Oct 30 21:47:16.997479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:16.997256    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5" event={"ID":"1ab33427-b274-4a08-86a6-a876c71057ab","Type":"ContainerDied","Data":"107ebd331d6be44a3480c1ab37b276b1e96c8bd7a96ce8a09291d593e319c854"}
Oct 30 21:47:18.157463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:18.157430    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:47:18.157888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:18.157486    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:47:18.204106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:18.204075    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:18.242999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:18.242969    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-xks-ca\") pod \"1ab33427-b274-4a08-86a6-a876c71057ab\" (UID: \"1ab33427-b274-4a08-86a6-a876c71057ab\") "
Oct 30 21:47:18.243244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:18.243034    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab33427-b274-4a08-86a6-a876c71057ab-kube-api-access-h9jvh\" (UniqueName: \"kubernetes.io/projected/1ab33427-b274-4a08-86a6-a876c71057ab-kube-api-access-h9jvh\") pod \"1ab33427-b274-4a08-86a6-a876c71057ab\" (UID: \"1ab33427-b274-4a08-86a6-a876c71057ab\") "
Oct 30 21:47:18.244071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:18.243367    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-openshift-ca\") pod \"1ab33427-b274-4a08-86a6-a876c71057ab\" (UID: \"1ab33427-b274-4a08-86a6-a876c71057ab\") "
Oct 30 21:47:18.244071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:18.243913    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-openshift-ca" pod "1ab33427-b274-4a08-86a6-a876c71057ab" (UID: "1ab33427-b274-4a08-86a6-a876c71057ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:18.244793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:18.244766    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:18.245755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:18.245735    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-xks-ca" pod "1ab33427-b274-4a08-86a6-a876c71057ab" (UID: "1ab33427-b274-4a08-86a6-a876c71057ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:18.245977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:18.245951    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab33427-b274-4a08-86a6-a876c71057ab-kube-api-access-h9jvh" pod "1ab33427-b274-4a08-86a6-a876c71057ab" (UID: "1ab33427-b274-4a08-86a6-a876c71057ab"). InnerVolumeSpecName "kube-api-access-h9jvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:18.391718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:18.391678    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9jvh\" (UniqueName: \"kubernetes.io/projected/1ab33427-b274-4a08-86a6-a876c71057ab-kube-api-access-h9jvh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:18.391718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:18.391719    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab33427-b274-4a08-86a6-a876c71057ab-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:19.016778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.016736    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"
Oct 30 21:47:19.016974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.016747    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5" event={"ID":"1ab33427-b274-4a08-86a6-a876c71057ab","Type":"ContainerDied","Data":"01fb6ec58f304c83358a2737f501f77b77a29f72785eaf8bc56c319020ba9e88"}
Oct 30 21:47:19.016974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.016881    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01fb6ec58f304c83358a2737f501f77b77a29f72785eaf8bc56c319020ba9e88"
Oct 30 21:47:19.405220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.405194    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:47:19.511027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.510993    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-socket\") pod \"f7574078-8dcf-4713-9574-c489ef73a943\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") "
Oct 30 21:47:19.511223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.511071    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-kube-api-access-zh56d\" (UniqueName: \"kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-kube-api-access-zh56d\") pod \"f7574078-8dcf-4713-9574-c489ef73a943\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") "
Oct 30 21:47:19.511223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.511124    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-istio-token\" (UniqueName: \"kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-istio-token\") pod \"f7574078-8dcf-4713-9574-c489ef73a943\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") "
Oct 30 21:47:19.511411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.511293    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-socket" pod "f7574078-8dcf-4713-9574-c489ef73a943" (UID: "f7574078-8dcf-4713-9574-c489ef73a943"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:19.511411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.511373    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7574078-8dcf-4713-9574-c489ef73a943-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7574078-8dcf-4713-9574-c489ef73a943-istio-podinfo\") pod \"f7574078-8dcf-4713-9574-c489ef73a943\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") "
Oct 30 21:47:19.511552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.511473    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-certs\") pod \"f7574078-8dcf-4713-9574-c489ef73a943\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") "
Oct 30 21:47:19.511552 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.511537    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7574078-8dcf-4713-9574-c489ef73a943-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7574078-8dcf-4713-9574-c489ef73a943-istiod-ca-cert\") pod \"f7574078-8dcf-4713-9574-c489ef73a943\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") "
Oct 30 21:47:19.511670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.511627    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-data\") pod \"f7574078-8dcf-4713-9574-c489ef73a943\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") "
Oct 30 21:47:19.511670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.511662    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-envoy\") pod \"f7574078-8dcf-4713-9574-c489ef73a943\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") "
Oct 30 21:47:19.511792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.511735    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-credential-socket\") pod \"f7574078-8dcf-4713-9574-c489ef73a943\" (UID: \"f7574078-8dcf-4713-9574-c489ef73a943\") "
Oct 30 21:47:19.512170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.512140    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-certs" pod "f7574078-8dcf-4713-9574-c489ef73a943" (UID: "f7574078-8dcf-4713-9574-c489ef73a943"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:19.512355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.512334    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-data" pod "f7574078-8dcf-4713-9574-c489ef73a943" (UID: "f7574078-8dcf-4713-9574-c489ef73a943"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:19.512939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.512912    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:19.513046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.512944    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:19.513046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.512961    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:19.513307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.513284    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7574078-8dcf-4713-9574-c489ef73a943-istiod-ca-cert" pod "f7574078-8dcf-4713-9574-c489ef73a943" (UID: "f7574078-8dcf-4713-9574-c489ef73a943"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:19.513589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.513565    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-credential-socket" pod "f7574078-8dcf-4713-9574-c489ef73a943" (UID: "f7574078-8dcf-4713-9574-c489ef73a943"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:19.514021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.513997    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-kube-api-access-zh56d" pod "f7574078-8dcf-4713-9574-c489ef73a943" (UID: "f7574078-8dcf-4713-9574-c489ef73a943"). InnerVolumeSpecName "kube-api-access-zh56d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:19.515055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.515030    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7574078-8dcf-4713-9574-c489ef73a943-istio-podinfo" pod "f7574078-8dcf-4713-9574-c489ef73a943" (UID: "f7574078-8dcf-4713-9574-c489ef73a943"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:47:19.515340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.515323    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-envoy" pod "f7574078-8dcf-4713-9574-c489ef73a943" (UID: "f7574078-8dcf-4713-9574-c489ef73a943"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:19.517563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.517543    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-istio-token" pod "f7574078-8dcf-4713-9574-c489ef73a943" (UID: "f7574078-8dcf-4713-9574-c489ef73a943"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:19.614486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.614453    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:19.614486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.614484    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7574078-8dcf-4713-9574-c489ef73a943-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:19.614486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.614493    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7574078-8dcf-4713-9574-c489ef73a943-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:19.614740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.614503    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:19.614740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.614512    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7574078-8dcf-4713-9574-c489ef73a943-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:19.614740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:19.614520    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh56d\" (UniqueName: \"kubernetes.io/projected/f7574078-8dcf-4713-9574-c489ef73a943-kube-api-access-zh56d\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:20.075416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:20.071665    4918 generic.go:356] "Generic (PLEG): container finished" podID="f7574078-8dcf-4713-9574-c489ef73a943" containerID="4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827" exitCode=0
Oct 30 21:47:20.075416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:20.072073    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" event={"ID":"f7574078-8dcf-4713-9574-c489ef73a943","Type":"ContainerDied","Data":"4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827"}
Oct 30 21:47:20.075416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:20.072117    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt" event={"ID":"f7574078-8dcf-4713-9574-c489ef73a943","Type":"ContainerDied","Data":"de31f84b03fdd59c740f810a918d9ac5daa4bf3ab26039d623a9f92dbe4f5edc"}
Oct 30 21:47:20.075416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:20.072143    4918 scope.go:122] "RemoveContainer" containerID="4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827"
Oct 30 21:47:20.075416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:20.072297    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"
Oct 30 21:47:20.165989 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:20.165693    4918 scope.go:122] "RemoveContainer" containerID="0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934"
Oct 30 21:47:20.188013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:20.186542    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"]
Oct 30 21:47:20.204483 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:20.199370    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-729802-openshift-default-6d76999bc7-29dgt"]
Oct 30 21:47:20.213675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:20.213653    4918 scope.go:122] "RemoveContainer" containerID="4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827"
Oct 30 21:47:20.215430 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:47:20.215297    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827\": container with ID starting with 4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827 not found: ID does not exist" containerID="4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827"
Oct 30 21:47:20.215430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:20.215328    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827"} err="failed to get container status \"4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827\": rpc error: code = NotFound desc = could not find container \"4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827\": container with ID starting with 4f2c702bb0c0ea1e5b8646fa6488f00acbb9a084fdf96ed23547f86b88c9e827 not found: ID does not exist"
Oct 30 21:47:20.215430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:20.215346    4918 scope.go:122] "RemoveContainer" containerID="0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934"
Oct 30 21:47:20.215821 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:47:20.215752    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934\": container with ID starting with 0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934 not found: ID does not exist" containerID="0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934"
Oct 30 21:47:20.215821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:20.215785    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934"} err="failed to get container status \"0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934\": rpc error: code = NotFound desc = could not find container \"0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934\": container with ID starting with 0cdec4fc5849270186c6a53ae87255b7384ffc1a48e904bc4044dc4899e6d934 not found: ID does not exist"
Oct 30 21:47:21.085106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:21.085075    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7574078-8dcf-4713-9574-c489ef73a943" path="/var/lib/kubelet/pods/f7574078-8dcf-4713-9574-c489ef73a943/volumes"
Oct 30 21:47:22.929001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:22.928972    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"
Oct 30 21:47:24.850593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:24.850564    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"]
Oct 30 21:47:29.269970 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:47:29.269923    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:47:29.851585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:29.851543    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4" podUID="18762808-b2c0-4c82-bda9-4b89c4a882c5" containerName="payload-processing" containerID="cri-o://661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4" gracePeriod=25
Oct 30 21:47:34.424280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:34.424249    4918 scope.go:122] "RemoveContainer" containerID="159853b42dd188278141b01f679abcb09186591e50a1a204ff619e3452ca6f60"
Oct 30 21:47:34.435732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:34.435713    4918 scope.go:122] "RemoveContainer" containerID="6fab9515308dda326ad38d9a5d6688f09b141d79807f9760530b093933f0ddfc"
Oct 30 21:47:34.446402 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:34.446371    4918 scope.go:122] "RemoveContainer" containerID="421037542ee20ce90506c74e4dfaa8d47d8ff9dc18f160784e060251d6d9dc3e"
Oct 30 21:47:38.369985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:38.369952    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4" podUID="18762808-b2c0-4c82-bda9-4b89c4a882c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:47:42.951231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.951196    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"]
Oct 30 21:47:42.953318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.953288    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy"
Oct 30 21:47:42.953318 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.953318    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy"
Oct 30 21:47:42.953758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.953430    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1ab33427-b274-4a08-86a6-a876c71057ab" containerName="revoke-keys"
Oct 30 21:47:42.953758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.953440    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab33427-b274-4a08-86a6-a876c71057ab" containerName="revoke-keys"
Oct 30 21:47:42.953758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.953456    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy"
Oct 30 21:47:42.953758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.953463    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy"
Oct 30 21:47:42.953758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.953673    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab33427-b274-4a08-86a6-a876c71057ab" containerName="revoke-keys"
Oct 30 21:47:42.953758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.953691    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy"
Oct 30 21:47:42.953758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.953709    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7574078-8dcf-4713-9574-c489ef73a943" containerName="istio-proxy"
Oct 30 21:47:42.965304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.965279    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:42.973324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.973300    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-87ae68-openshift-default-dockercfg-sk5qm\""
Oct 30 21:47:42.974513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:42.974485    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"]
Oct 30 21:47:43.128515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.126670    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrlw\" (UniqueName: \"kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-kube-api-access-7wrlw\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.128515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.126792    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-podinfo\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.128515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.126930    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-data\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.128515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.127008    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-envoy\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.128515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.127056    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-certs\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.128515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.127130    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istiod-ca-cert\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.128515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.127228    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-token\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.128515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.127288    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-socket\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.128515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.127475    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-credential-socket\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.252056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.252007    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-token\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.252922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.252880    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-socket\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.254074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.254052    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-credential-socket\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.256362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.256341    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrlw\" (UniqueName: \"kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-kube-api-access-7wrlw\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.257079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.257059    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-podinfo\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.258861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.253764    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-socket\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.258994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.256234    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-credential-socket\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.259157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.259141    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-data\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.259569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.259553    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-envoy\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.259999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.259983    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-certs\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.260458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.260441    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istiod-ca-cert\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.261373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.259509    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-data\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.261542 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.261133    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istiod-ca-cert\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.261643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.260312    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-certs\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.283948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.283694    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-envoy\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.309110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.305122    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-token\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.378311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.371157    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrlw\" (UniqueName: \"kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-kube-api-access-7wrlw\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.421414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.408156    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-podinfo\") pod \"e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") " pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.588047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.587973    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:43.913446 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:47:43.913413    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75e2a4e_7c27_4a3f_8cc0_09478b8dfa59.slice/crio-3aaac4eaa6de48aa2bbf681434509d61aa1d6fb0359eb2f9e912be8a315d7ee7 WatchSource:0}: Error finding container 3aaac4eaa6de48aa2bbf681434509d61aa1d6fb0359eb2f9e912be8a315d7ee7: Status 404 returned error can't find the container with id 3aaac4eaa6de48aa2bbf681434509d61aa1d6fb0359eb2f9e912be8a315d7ee7
Oct 30 21:47:43.914990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.914961    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"]
Oct 30 21:47:43.916851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.916820    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:47:43.916932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.916886    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:47:43.916932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:43.916916    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:47:44.495032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:44.494994    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" event={"ID":"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59","Type":"ContainerStarted","Data":"acf0d574c3e6a38883f1b3712e0dd3bd6c8d5a308f896a051e7068e28d4ada50"}
Oct 30 21:47:44.495513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:44.495094    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" event={"ID":"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59","Type":"ContainerStarted","Data":"3aaac4eaa6de48aa2bbf681434509d61aa1d6fb0359eb2f9e912be8a315d7ee7"}
Oct 30 21:47:44.524211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:44.524163    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" podStartSLOduration=2.524150003 podStartE2EDuration="2.524150003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:44.520240134 +0000 UTC m=+2777.566799297" watchObservedRunningTime="2026-10-30 21:47:44.524150003 +0000 UTC m=+2777.570709148"
Oct 30 21:47:44.588123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:44.588081    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:45.595835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:45.595800    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:46.512994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:46.512956    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:46.514130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:46.514110    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:47:47.284418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:47.281546    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"]
Oct 30 21:47:47.324417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:47.323767    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7b5a7c-m49l5"]
Oct 30 21:47:48.373437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:48.373353    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4" podUID="18762808-b2c0-4c82-bda9-4b89c4a882c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:47:49.088123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:49.088084    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab33427-b274-4a08-86a6-a876c71057ab" path="/var/lib/kubelet/pods/1ab33427-b274-4a08-86a6-a876c71057ab/volumes"
Oct 30 21:47:50.151165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.151143    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"
Oct 30 21:47:50.235693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.235654    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18762808-b2c0-4c82-bda9-4b89c4a882c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18762808-b2c0-4c82-bda9-4b89c4a882c5-plugins-config-volume\") pod \"18762808-b2c0-4c82-bda9-4b89c4a882c5\" (UID: \"18762808-b2c0-4c82-bda9-4b89c4a882c5\") "
Oct 30 21:47:50.235966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.235953    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18762808-b2c0-4c82-bda9-4b89c4a882c5-kube-api-access-xnm29\" (UniqueName: \"kubernetes.io/projected/18762808-b2c0-4c82-bda9-4b89c4a882c5-kube-api-access-xnm29\") pod \"18762808-b2c0-4c82-bda9-4b89c4a882c5\" (UID: \"18762808-b2c0-4c82-bda9-4b89c4a882c5\") "
Oct 30 21:47:50.236038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.236013    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18762808-b2c0-4c82-bda9-4b89c4a882c5-plugins-config-volume" pod "18762808-b2c0-4c82-bda9-4b89c4a882c5" (UID: "18762808-b2c0-4c82-bda9-4b89c4a882c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:50.237102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.237077    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18762808-b2c0-4c82-bda9-4b89c4a882c5-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:50.239085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.239057    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18762808-b2c0-4c82-bda9-4b89c4a882c5-kube-api-access-xnm29" pod "18762808-b2c0-4c82-bda9-4b89c4a882c5" (UID: "18762808-b2c0-4c82-bda9-4b89c4a882c5"). InnerVolumeSpecName "kube-api-access-xnm29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:50.370463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.341640    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnm29\" (UniqueName: \"kubernetes.io/projected/18762808-b2c0-4c82-bda9-4b89c4a882c5-kube-api-access-xnm29\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:50.591136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.591098    4918 generic.go:356] "Generic (PLEG): container finished" podID="18762808-b2c0-4c82-bda9-4b89c4a882c5" containerID="661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4" exitCode=0
Oct 30 21:47:50.591337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.591153    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"
Oct 30 21:47:50.591337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.591175    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4" event={"ID":"18762808-b2c0-4c82-bda9-4b89c4a882c5","Type":"ContainerDied","Data":"661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4"}
Oct 30 21:47:50.591337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.591230    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4" event={"ID":"18762808-b2c0-4c82-bda9-4b89c4a882c5","Type":"ContainerDied","Data":"24d990f163b3cd6331259010d908711a24e87c3b8bb602d44d2c9fb067ceaadc"}
Oct 30 21:47:50.591337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.591258    4918 scope.go:122] "RemoveContainer" containerID="661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4"
Oct 30 21:47:50.603798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.603780    4918 scope.go:122] "RemoveContainer" containerID="661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4"
Oct 30 21:47:50.604051 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:47:50.604032    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4\": container with ID starting with 661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4 not found: ID does not exist" containerID="661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4"
Oct 30 21:47:50.604123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.604057    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4"} err="failed to get container status \"661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4\": rpc error: code = NotFound desc = could not find container \"661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4\": container with ID starting with 661aae95b6e8265ad163cbb0fa65b187a60b88ee5b3e85b1c4fd0c48f7808aa4 not found: ID does not exist"
Oct 30 21:47:50.621723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.621515    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"]
Oct 30 21:47:50.627594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:50.627569    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7b5a7c-7bdffd87b7-ptgz4"]
Oct 30 21:47:51.084996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:51.084966    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18762808-b2c0-4c82-bda9-4b89c4a882c5" path="/var/lib/kubelet/pods/18762808-b2c0-4c82-bda9-4b89c4a882c5/volumes"
Oct 30 21:47:55.502416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:55.477238    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"]
Oct 30 21:47:55.575417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:55.543691    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy" containerID="cri-o://2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf" gracePeriod=30
Oct 30 21:47:55.703440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:55.693148    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:55.864665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:55.864564    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:55.929279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:55.927161    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:47:55.931086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:55.931046    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:56.025413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.022169    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Oct 30 21:47:56.040425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.039952    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:56.061424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.059037    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:47:56.061424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.060116    4918 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Oct 30 21:47:56.180193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.178620    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:56.342417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.341691    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"]
Oct 30 21:47:56.352740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.343148    4918 cpu_manager.go:414] "containerMap: removing container" podUID="18762808-b2c0-4c82-bda9-4b89c4a882c5" containerName="payload-processing"
Oct 30 21:47:56.352740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.343174    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18762808-b2c0-4c82-bda9-4b89c4a882c5" containerName="payload-processing"
Oct 30 21:47:56.361618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.359607    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="18762808-b2c0-4c82-bda9-4b89c4a882c5" containerName="payload-processing"
Oct 30 21:47:56.364926 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.364901    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:47:56.477519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.477491    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:47:56.506882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.502845    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:47:56.514416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.509961    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:47:56.548415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.533838    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:47:56.548415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.544901    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2\" (UID: \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:47:56.548415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.545085    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r95gl\" (UniqueName: \"kubernetes.io/projected/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-kube-api-access-r95gl\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2\" (UID: \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:47:56.548415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.545196    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2\" (UID: \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:47:56.651434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.651379    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2\" (UID: \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:47:56.661031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.661000    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r95gl\" (UniqueName: \"kubernetes.io/projected/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-kube-api-access-r95gl\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2\" (UID: \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:47:56.661351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.661331    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2\" (UID: \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:47:56.680257 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.680224    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2\" (UID: \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:47:56.812464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.810381    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2\" (UID: \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:47:56.812464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.810495    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"]
Oct 30 21:47:56.813344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.813315    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r95gl\" (UniqueName: \"kubernetes.io/projected/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-kube-api-access-r95gl\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2\" (UID: \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:47:56.968375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.958669    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:57.040365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:56.970153    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 21:47:57.043425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.041493    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:47:57.074496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.062490    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:57.074496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.062903    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:47:57.098457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.098418    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:47:57.115879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.115717    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded"
Oct 30 21:47:57.116169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.116132    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 21:47:57.169649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.160078    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq" podUID="1ec8e779-955f-4570-81ad-480d8cefca7a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:47:57.294431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.265330    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:47:57.542883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.535913    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:47:57.542883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.535975    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:47:57.614164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.614134    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:47:57.614492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.614476    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:47:57.614695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.614682    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:47:57.879767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.660464    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:47:57.880074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.880054    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:47:57.880213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.880203    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:47:57.884611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.884581    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq"
Oct 30 21:47:57.884883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:57.884867    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:47:58.355616 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:47:58.355589    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49d9a1d_ec9d_4cf2_9c68_723a2982a9eb.slice/crio-039876c5c8854b2b61be2dd2aaea70e59fc4b567e0f13ce23afb361bc53435d4 WatchSource:0}: Error finding container 039876c5c8854b2b61be2dd2aaea70e59fc4b567e0f13ce23afb361bc53435d4: Status 404 returned error can't find the container with id 039876c5c8854b2b61be2dd2aaea70e59fc4b567e0f13ce23afb361bc53435d4
Oct 30 21:47:58.358143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:58.358119    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"]
Oct 30 21:47:58.669239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:58.669159    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2" event={"ID":"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb","Type":"ContainerStarted","Data":"039876c5c8854b2b61be2dd2aaea70e59fc4b567e0f13ce23afb361bc53435d4"}
Oct 30 21:47:59.161293 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:47:59.161254    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:47:59.732436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:47:59.732314    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2" event={"ID":"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb","Type":"ContainerStarted","Data":"122ef3ba4778fe9d7a9579d730e3d9af322236bc4645f830bca5aa379e82edf4"}
Oct 30 21:48:00.743121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:00.743089    4918 generic.go:356] "Generic (PLEG): container finished" podID="e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb" containerID="122ef3ba4778fe9d7a9579d730e3d9af322236bc4645f830bca5aa379e82edf4" exitCode=0
Oct 30 21:48:00.743685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:00.743180    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2" event={"ID":"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb","Type":"ContainerDied","Data":"122ef3ba4778fe9d7a9579d730e3d9af322236bc4645f830bca5aa379e82edf4"}
Oct 30 21:48:01.017127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.017094    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:48:01.093017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.092984    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-kube-api-access-r95gl\" (UniqueName: \"kubernetes.io/projected/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-kube-api-access-r95gl\") pod \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\" (UID: \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\") "
Oct 30 21:48:01.093251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.093226    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-openshift-ca\") pod \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\" (UID: \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\") "
Oct 30 21:48:01.093591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.093568    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-xks-ca\") pod \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\" (UID: \"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb\") "
Oct 30 21:48:01.095083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.094438    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-openshift-ca" pod "e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb" (UID: "e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:01.101731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.101706    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-xks-ca" pod "e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb" (UID: "e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:01.104838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.104816    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-kube-api-access-r95gl" pod "e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb" (UID: "e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb"). InnerVolumeSpecName "kube-api-access-r95gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:01.143634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.143609    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:48:01.240477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.233195    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r95gl\" (UniqueName: \"kubernetes.io/projected/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-kube-api-access-r95gl\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.240477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.233236    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.240477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.233252    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.334855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.334777    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-token\" (UniqueName: \"kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-token\") pod \"f30e391e-32ba-492a-98af-ac3c4d6089d1\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") "
Oct 30 21:48:01.335008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.334892    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-kube-api-access-s4ck4\" (UniqueName: \"kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-kube-api-access-s4ck4\") pod \"f30e391e-32ba-492a-98af-ac3c4d6089d1\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") "
Oct 30 21:48:01.335008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.334964    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-socket\") pod \"f30e391e-32ba-492a-98af-ac3c4d6089d1\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") "
Oct 30 21:48:01.335129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.335065    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30e391e-32ba-492a-98af-ac3c4d6089d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30e391e-32ba-492a-98af-ac3c4d6089d1-istiod-ca-cert\") pod \"f30e391e-32ba-492a-98af-ac3c4d6089d1\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") "
Oct 30 21:48:01.335193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.335128    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-data\") pod \"f30e391e-32ba-492a-98af-ac3c4d6089d1\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") "
Oct 30 21:48:01.335258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.335215    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-podinfo\") pod \"f30e391e-32ba-492a-98af-ac3c4d6089d1\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") "
Oct 30 21:48:01.335323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.335300    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-credential-socket\") pod \"f30e391e-32ba-492a-98af-ac3c4d6089d1\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") "
Oct 30 21:48:01.335445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.335418    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-envoy\") pod \"f30e391e-32ba-492a-98af-ac3c4d6089d1\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") "
Oct 30 21:48:01.335628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.335293    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-socket" pod "f30e391e-32ba-492a-98af-ac3c4d6089d1" (UID: "f30e391e-32ba-492a-98af-ac3c4d6089d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:01.335767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.335586    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-credential-socket" pod "f30e391e-32ba-492a-98af-ac3c4d6089d1" (UID: "f30e391e-32ba-492a-98af-ac3c4d6089d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:01.335767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.335638    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-certs\") pod \"f30e391e-32ba-492a-98af-ac3c4d6089d1\" (UID: \"f30e391e-32ba-492a-98af-ac3c4d6089d1\") "
Oct 30 21:48:01.335767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.335644    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-data" pod "f30e391e-32ba-492a-98af-ac3c4d6089d1" (UID: "f30e391e-32ba-492a-98af-ac3c4d6089d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:01.336005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.335963    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-certs" pod "f30e391e-32ba-492a-98af-ac3c4d6089d1" (UID: "f30e391e-32ba-492a-98af-ac3c4d6089d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:01.336082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.336049    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30e391e-32ba-492a-98af-ac3c4d6089d1-istiod-ca-cert" pod "f30e391e-32ba-492a-98af-ac3c4d6089d1" (UID: "f30e391e-32ba-492a-98af-ac3c4d6089d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:01.336558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.336535    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.336647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.336562    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.336647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.336576    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.336647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.336588    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30e391e-32ba-492a-98af-ac3c4d6089d1-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.336647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.336601    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.337338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.337318    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-token" pod "f30e391e-32ba-492a-98af-ac3c4d6089d1" (UID: "f30e391e-32ba-492a-98af-ac3c4d6089d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:01.338049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.338020    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-podinfo" pod "f30e391e-32ba-492a-98af-ac3c4d6089d1" (UID: "f30e391e-32ba-492a-98af-ac3c4d6089d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:48:01.338138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.338055    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-envoy" pod "f30e391e-32ba-492a-98af-ac3c4d6089d1" (UID: "f30e391e-32ba-492a-98af-ac3c4d6089d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:01.338138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.338063    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-kube-api-access-s4ck4" pod "f30e391e-32ba-492a-98af-ac3c4d6089d1" (UID: "f30e391e-32ba-492a-98af-ac3c4d6089d1"). InnerVolumeSpecName "kube-api-access-s4ck4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:01.438475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.438442    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.438475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.438469    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.438475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.438480    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ck4\" (UniqueName: \"kubernetes.io/projected/f30e391e-32ba-492a-98af-ac3c4d6089d1-kube-api-access-s4ck4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.438700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.438490    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30e391e-32ba-492a-98af-ac3c4d6089d1-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:01.756124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.756090    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"
Oct 30 21:48:01.756658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.756094    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2" event={"ID":"e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb","Type":"ContainerDied","Data":"039876c5c8854b2b61be2dd2aaea70e59fc4b567e0f13ce23afb361bc53435d4"}
Oct 30 21:48:01.756658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.756227    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="039876c5c8854b2b61be2dd2aaea70e59fc4b567e0f13ce23afb361bc53435d4"
Oct 30 21:48:01.757911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.757867    4918 generic.go:356] "Generic (PLEG): container finished" podID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerID="2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf" exitCode=0
Oct 30 21:48:01.758081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.757933    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"
Oct 30 21:48:01.758081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.757941    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" event={"ID":"f30e391e-32ba-492a-98af-ac3c4d6089d1","Type":"ContainerDied","Data":"2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf"}
Oct 30 21:48:01.758081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.757998    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx" event={"ID":"f30e391e-32ba-492a-98af-ac3c4d6089d1","Type":"ContainerDied","Data":"6c10569304d6d4493b060b8cc6fef79da7ac1bf844099f10b209f79c5a693afb"}
Oct 30 21:48:01.758081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.758028    4918 scope.go:122] "RemoveContainer" containerID="2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf"
Oct 30 21:48:01.778924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.778905    4918 scope.go:122] "RemoveContainer" containerID="2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf"
Oct 30 21:48:01.779272 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:48:01.779251    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf\": container with ID starting with 2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf not found: ID does not exist" containerID="2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf"
Oct 30 21:48:01.779361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.779283    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf"} err="failed to get container status \"2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf\": rpc error: code = NotFound desc = could not find container \"2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf\": container with ID starting with 2e036fe167d14571fe1a690fdd71f9ec4b926557c4312b9b4abfd68301adaccf not found: ID does not exist"
Oct 30 21:48:01.800058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.800035    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"]
Oct 30 21:48:01.808450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:01.808425    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7b5a7c-openshift-default-c94cf785c-6h8gx"]
Oct 30 21:48:03.085020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:03.084984    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" path="/var/lib/kubelet/pods/f30e391e-32ba-492a-98af-ac3c4d6089d1/volumes"
Oct 30 21:48:06.983565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:06.983525    4918 generic.go:356] "Generic (PLEG): container finished" podID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerID="7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2" exitCode=0
Oct 30 21:48:06.984146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:06.983607    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" event={"ID":"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c","Type":"ContainerDied","Data":"7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2"}
Oct 30 21:48:06.984216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:06.984203    4918 scope.go:122] "RemoveContainer" containerID="7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2"
Oct 30 21:48:07.003181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.003133    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:48:07.010467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.003514    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:48:07.010800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.010768    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:48:07.583861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.583827    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"]
Oct 30 21:48:07.584916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.584891    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb" containerName="revoke-keys"
Oct 30 21:48:07.584916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.584913    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb" containerName="revoke-keys"
Oct 30 21:48:07.585121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.584985    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy"
Oct 30 21:48:07.585121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.584995    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy"
Oct 30 21:48:07.585250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.585233    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb" containerName="revoke-keys"
Oct 30 21:48:07.585310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.585273    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30e391e-32ba-492a-98af-ac3c4d6089d1" containerName="istio-proxy"
Oct 30 21:48:07.589591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.589564    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.598884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.598864    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-001667-openshift-default-dockercfg-phvhn\""
Oct 30 21:48:07.610808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.610779    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"]
Oct 30 21:48:07.678559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.678527    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-socket\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.678559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.678564    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-podinfo\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.678836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.678777    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-credential-socket\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.678931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.678908    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istiod-ca-cert\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.679093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.679010    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjlh\" (UniqueName: \"kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-kube-api-access-jrjlh\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.679093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.679076    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-envoy\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.679268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.679246    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-data\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.679371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.679298    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-certs\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.679371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.679328    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-token\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.809416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.797976    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-credential-socket\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.809416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.798146    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istiod-ca-cert\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.809416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.798180    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjlh\" (UniqueName: \"kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-kube-api-access-jrjlh\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.809416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.798256    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-envoy\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.809416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.798455    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-data\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.809416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.798520    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-certs\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.809416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.798549    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-token\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.809416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.798577    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-socket\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.809416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.798601    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-podinfo\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.809416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.802914    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istiod-ca-cert\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.809416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.807088    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-credential-socket\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.850460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.836563    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-envoy\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.850460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.836627    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-certs\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.850460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.837152    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-data\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.850460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.837417    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-socket\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.859529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.859072    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-token\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.883417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.873282    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjlh\" (UniqueName: \"kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-kube-api-access-jrjlh\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.889412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.884376    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-podinfo\") pod \"e2e-inf-b-001667-openshift-default-c686545cc-97cd8\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") " pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:07.916467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:07.908803    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:08.103187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:08.102025    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" event={"ID":"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c","Type":"ContainerStarted","Data":"6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa"}
Oct 30 21:48:08.519418 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:48:08.519368    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119bae8a_d416_4701_bfc6_eeabc9db6ef9.slice/crio-c397268bad3fc2fa1c401eb35f22297ec6ed8b587f6eb25d5ced202aeaa0a701 WatchSource:0}: Error finding container c397268bad3fc2fa1c401eb35f22297ec6ed8b587f6eb25d5ced202aeaa0a701: Status 404 returned error can't find the container with id c397268bad3fc2fa1c401eb35f22297ec6ed8b587f6eb25d5ced202aeaa0a701
Oct 30 21:48:08.519988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:08.519965    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"]
Oct 30 21:48:08.522661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:08.522627    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:48:08.522778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:08.522694    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:48:08.522778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:08.522721    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:48:08.812469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:08.804578    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:48:08.812469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:08.805963    4918 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:48:08.812469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:08.806017    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused"
Oct 30 21:48:09.125503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:09.125375    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" event={"ID":"119bae8a-d416-4701-bfc6-eeabc9db6ef9","Type":"ContainerStarted","Data":"8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf"}
Oct 30 21:48:09.125503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:09.125454    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" event={"ID":"119bae8a-d416-4701-bfc6-eeabc9db6ef9","Type":"ContainerStarted","Data":"c397268bad3fc2fa1c401eb35f22297ec6ed8b587f6eb25d5ced202aeaa0a701"}
Oct 30 21:48:09.147149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:09.147091    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" podStartSLOduration=2.147072797 podStartE2EDuration="2.147072797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:09.144896562 +0000 UTC m=+2802.191455722" watchObservedRunningTime="2026-10-30 21:48:09.147072797 +0000 UTC m=+2802.193631944"
Oct 30 21:48:09.807415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:09.807324    4918 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:48:09.807629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:09.807412    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused"
Oct 30 21:48:09.922485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:09.916581    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:09.943475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:09.941968    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:10.135015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:10.134919    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:10.136095 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:10.136071    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:48:10.457595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:10.457511    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:48:10.804418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:10.801781    4918 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:48:10.804418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:10.801848    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused"
Oct 30 21:48:11.801782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:11.801746    4918 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:48:11.802809 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:11.802776    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused"
Oct 30 21:48:12.796425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:12.795820    4918 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:48:12.796425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:12.795900    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused"
Oct 30 21:48:13.067800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.067725    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"]
Oct 30 21:48:13.073721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.073695    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"
Oct 30 21:48:13.082727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.082695    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-001667\""
Oct 30 21:48:13.082921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.082821    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-001667-dockercfg-9n5vp\""
Oct 30 21:48:13.135208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.135177    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"]
Oct 30 21:48:13.144292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.144263    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"]
Oct 30 21:48:13.151091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.151066    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"
Oct 30 21:48:13.317973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.302006    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc9bn\" (UniqueName: \"kubernetes.io/projected/3568958b-8717-48bb-8dae-272d42ba8117-kube-api-access-sc9bn\") pod \"payload-processing-e2e-inf-b-001667-55b68dfd47-827xj\" (UID: \"3568958b-8717-48bb-8dae-272d42ba8117\") " pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"
Oct 30 21:48:13.317973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.302129    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl\" (UID: \"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"
Oct 30 21:48:13.317973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.302528    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3568958b-8717-48bb-8dae-272d42ba8117-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-001667-55b68dfd47-827xj\" (UID: \"3568958b-8717-48bb-8dae-272d42ba8117\") " pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"
Oct 30 21:48:13.317973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.302593    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqrzg\" (UniqueName: \"kubernetes.io/projected/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-kube-api-access-vqrzg\") pod \"payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl\" (UID: \"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"
Oct 30 21:48:13.506455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.493641    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3568958b-8717-48bb-8dae-272d42ba8117-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-001667-55b68dfd47-827xj\" (UID: \"3568958b-8717-48bb-8dae-272d42ba8117\") " pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"
Oct 30 21:48:13.506455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.493713    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqrzg\" (UniqueName: \"kubernetes.io/projected/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-kube-api-access-vqrzg\") pod \"payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl\" (UID: \"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"
Oct 30 21:48:13.506455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.493945    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9bn\" (UniqueName: \"kubernetes.io/projected/3568958b-8717-48bb-8dae-272d42ba8117-kube-api-access-sc9bn\") pod \"payload-processing-e2e-inf-b-001667-55b68dfd47-827xj\" (UID: \"3568958b-8717-48bb-8dae-272d42ba8117\") " pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"
Oct 30 21:48:13.506455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.494022    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl\" (UID: \"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"
Oct 30 21:48:13.506455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.495015    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl\" (UID: \"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"
Oct 30 21:48:13.536890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.517008    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3568958b-8717-48bb-8dae-272d42ba8117-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-001667-55b68dfd47-827xj\" (UID: \"3568958b-8717-48bb-8dae-272d42ba8117\") " pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"
Oct 30 21:48:13.718528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.696230    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"]
Oct 30 21:48:13.732763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.732226    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9bn\" (UniqueName: \"kubernetes.io/projected/3568958b-8717-48bb-8dae-272d42ba8117-kube-api-access-sc9bn\") pod \"payload-processing-e2e-inf-b-001667-55b68dfd47-827xj\" (UID: \"3568958b-8717-48bb-8dae-272d42ba8117\") " pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"
Oct 30 21:48:13.734291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.734269    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqrzg\" (UniqueName: \"kubernetes.io/projected/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-kube-api-access-vqrzg\") pod \"payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl\" (UID: \"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"
Oct 30 21:48:13.812504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.812469    4918 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused" start-of-body=
Oct 30 21:48:13.812677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.812549    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.129:15021/healthz/ready\": dial tcp 10.132.1.129:15021: connect: connection refused"
Oct 30 21:48:13.839556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:13.839528    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"
Oct 30 21:48:14.029248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.029219    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"
Oct 30 21:48:14.178323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.178291    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"]
Oct 30 21:48:14.183760 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:48:14.183707    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6cc2dd_df3e_4fa7_a212_a04bb37e5155.slice/crio-693f378edfd586607546efd8e09e317d40e1f721f0c272ee3283043968de72dc WatchSource:0}: Error finding container 693f378edfd586607546efd8e09e317d40e1f721f0c272ee3283043968de72dc: Status 404 returned error can't find the container with id 693f378edfd586607546efd8e09e317d40e1f721f0c272ee3283043968de72dc
Oct 30 21:48:14.445735 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:48:14.445706    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3568958b_8717_48bb_8dae_272d42ba8117.slice/crio-ef143b7e9f8a55e470d77d4112428f8782c62b3bbc585b44af023f59bf5cf884 WatchSource:0}: Error finding container ef143b7e9f8a55e470d77d4112428f8782c62b3bbc585b44af023f59bf5cf884: Status 404 returned error can't find the container with id ef143b7e9f8a55e470d77d4112428f8782c62b3bbc585b44af023f59bf5cf884
Oct 30 21:48:14.446097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.446075    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"]
Oct 30 21:48:14.705742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.705639    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl" event={"ID":"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155","Type":"ContainerStarted","Data":"41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9"}
Oct 30 21:48:14.705742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.705694    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl" event={"ID":"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155","Type":"ContainerStarted","Data":"693f378edfd586607546efd8e09e317d40e1f721f0c272ee3283043968de72dc"}
Oct 30 21:48:14.705742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.705737    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"
Oct 30 21:48:14.707683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.707658    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj" event={"ID":"3568958b-8717-48bb-8dae-272d42ba8117","Type":"ContainerStarted","Data":"af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d"}
Oct 30 21:48:14.707826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.707703    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj" event={"ID":"3568958b-8717-48bb-8dae-272d42ba8117","Type":"ContainerStarted","Data":"ef143b7e9f8a55e470d77d4112428f8782c62b3bbc585b44af023f59bf5cf884"}
Oct 30 21:48:14.707826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.707787    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"
Oct 30 21:48:14.736072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.736011    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl" podStartSLOduration=1.735990887 podStartE2EDuration="1.735990887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:14.728813451 +0000 UTC m=+2807.775372624" watchObservedRunningTime="2026-10-30 21:48:14.735990887 +0000 UTC m=+2807.782550033"
Oct 30 21:48:14.752351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.752284    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj" podStartSLOduration=1.752269601 podStartE2EDuration="1.752269601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:14.751838265 +0000 UTC m=+2807.798397410" watchObservedRunningTime="2026-10-30 21:48:14.752269601 +0000 UTC m=+2807.798828743"
Oct 30 21:48:14.798478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.798447    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:48:14.799625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:14.799599    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:48:24.115487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:24.114438    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"]
Oct 30 21:48:24.195418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:24.193247    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b30a42-m79n2"]
Oct 30 21:48:25.080971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:25.080939    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb" path="/var/lib/kubelet/pods/e49d9a1d-ec9d-4cf2-9c68-723a2982a9eb/volumes"
Oct 30 21:48:25.722352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:25.722315    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"
Oct 30 21:48:25.722885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:25.722745    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"
Oct 30 21:48:29.162377 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:48:29.162339    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:48:32.139581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.139541    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"]
Oct 30 21:48:32.140281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.139802    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy" containerID="cri-o://874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8" gracePeriod=30
Oct 30 21:48:32.633825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.633782    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"]
Oct 30 21:48:32.640125 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.640107    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:32.648769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.648744    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:48:32.648923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.648809    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:48:32.652528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.652507    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"]
Oct 30 21:48:32.666527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.666493    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9581fcef-e655-4006-8561-7643c3852e01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8\" (UID: \"9581fcef-e655-4006-8561-7643c3852e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:32.666908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.666881    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9581fcef-e655-4006-8561-7643c3852e01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8\" (UID: \"9581fcef-e655-4006-8561-7643c3852e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:32.667543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.667510    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jklbs\" (UniqueName: \"kubernetes.io/projected/9581fcef-e655-4006-8561-7643c3852e01-kube-api-access-jklbs\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8\" (UID: \"9581fcef-e655-4006-8561-7643c3852e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:32.737965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.737928    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:48:32.738135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.737988    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:48:32.769330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.769303    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jklbs\" (UniqueName: \"kubernetes.io/projected/9581fcef-e655-4006-8561-7643c3852e01-kube-api-access-jklbs\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8\" (UID: \"9581fcef-e655-4006-8561-7643c3852e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:32.769496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.769453    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9581fcef-e655-4006-8561-7643c3852e01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8\" (UID: \"9581fcef-e655-4006-8561-7643c3852e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:32.769680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.769662    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9581fcef-e655-4006-8561-7643c3852e01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8\" (UID: \"9581fcef-e655-4006-8561-7643c3852e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:32.770178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.770150    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9581fcef-e655-4006-8561-7643c3852e01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8\" (UID: \"9581fcef-e655-4006-8561-7643c3852e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:32.772221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.772197    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9581fcef-e655-4006-8561-7643c3852e01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8\" (UID: \"9581fcef-e655-4006-8561-7643c3852e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:32.779767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.779749    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jklbs\" (UniqueName: \"kubernetes.io/projected/9581fcef-e655-4006-8561-7643c3852e01-kube-api-access-jklbs\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8\" (UID: \"9581fcef-e655-4006-8561-7643c3852e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:32.959716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:32.959626    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:33.259118 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:48:33.259085    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9581fcef_e655_4006_8561_7643c3852e01.slice/crio-a73aaab0dc127afef693d22b535b7e180f95dadc3cbab4233c53a334c45171be WatchSource:0}: Error finding container a73aaab0dc127afef693d22b535b7e180f95dadc3cbab4233c53a334c45171be: Status 404 returned error can't find the container with id a73aaab0dc127afef693d22b535b7e180f95dadc3cbab4233c53a334c45171be
Oct 30 21:48:33.260364 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:33.260336    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"]
Oct 30 21:48:34.054071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:34.054032    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8" event={"ID":"9581fcef-e655-4006-8561-7643c3852e01","Type":"ContainerStarted","Data":"c0ccec3337452b02d1f27f765aa2f38a30d928d77250cdf8a38305e3d1f9b1ef"}
Oct 30 21:48:34.054275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:34.054095    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8" event={"ID":"9581fcef-e655-4006-8561-7643c3852e01","Type":"ContainerStarted","Data":"a73aaab0dc127afef693d22b535b7e180f95dadc3cbab4233c53a334c45171be"}
Oct 30 21:48:34.075889 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:34.075831    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8" podStartSLOduration=2.075814571 podStartE2EDuration="2.075814571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:34.072869519 +0000 UTC m=+2827.119428674" watchObservedRunningTime="2026-10-30 21:48:34.075814571 +0000 UTC m=+2827.122373717"
Oct 30 21:48:35.064818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:35.064779    4918 generic.go:356] "Generic (PLEG): container finished" podID="9581fcef-e655-4006-8561-7643c3852e01" containerID="c0ccec3337452b02d1f27f765aa2f38a30d928d77250cdf8a38305e3d1f9b1ef" exitCode=0
Oct 30 21:48:35.065292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:35.064828    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8" event={"ID":"9581fcef-e655-4006-8561-7643c3852e01","Type":"ContainerDied","Data":"c0ccec3337452b02d1f27f765aa2f38a30d928d77250cdf8a38305e3d1f9b1ef"}
Oct 30 21:48:36.274330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:36.274302    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:36.309022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:36.308994    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9581fcef-e655-4006-8561-7643c3852e01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9581fcef-e655-4006-8561-7643c3852e01-maas-api-openshift-ca\") pod \"9581fcef-e655-4006-8561-7643c3852e01\" (UID: \"9581fcef-e655-4006-8561-7643c3852e01\") "
Oct 30 21:48:36.309289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:36.309275    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9581fcef-e655-4006-8561-7643c3852e01-kube-api-access-jklbs\" (UniqueName: \"kubernetes.io/projected/9581fcef-e655-4006-8561-7643c3852e01-kube-api-access-jklbs\") pod \"9581fcef-e655-4006-8561-7643c3852e01\" (UID: \"9581fcef-e655-4006-8561-7643c3852e01\") "
Oct 30 21:48:36.309343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:36.309307    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9581fcef-e655-4006-8561-7643c3852e01-maas-api-openshift-ca" pod "9581fcef-e655-4006-8561-7643c3852e01" (UID: "9581fcef-e655-4006-8561-7643c3852e01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:36.309422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:36.309343    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9581fcef-e655-4006-8561-7643c3852e01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9581fcef-e655-4006-8561-7643c3852e01-maas-api-xks-ca\") pod \"9581fcef-e655-4006-8561-7643c3852e01\" (UID: \"9581fcef-e655-4006-8561-7643c3852e01\") "
Oct 30 21:48:36.309940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:36.309909    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9581fcef-e655-4006-8561-7643c3852e01-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:36.311631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:36.311610    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9581fcef-e655-4006-8561-7643c3852e01-kube-api-access-jklbs" pod "9581fcef-e655-4006-8561-7643c3852e01" (UID: "9581fcef-e655-4006-8561-7643c3852e01"). InnerVolumeSpecName "kube-api-access-jklbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:36.311631 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:36.311616    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9581fcef-e655-4006-8561-7643c3852e01-maas-api-xks-ca" pod "9581fcef-e655-4006-8561-7643c3852e01" (UID: "9581fcef-e655-4006-8561-7643c3852e01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:36.411248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:36.411215    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9581fcef-e655-4006-8561-7643c3852e01-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:36.411248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:36.411243    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jklbs\" (UniqueName: \"kubernetes.io/projected/9581fcef-e655-4006-8561-7643c3852e01-kube-api-access-jklbs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.084092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.084015    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"
Oct 30 21:48:37.084092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.084039    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8" event={"ID":"9581fcef-e655-4006-8561-7643c3852e01","Type":"ContainerDied","Data":"a73aaab0dc127afef693d22b535b7e180f95dadc3cbab4233c53a334c45171be"}
Oct 30 21:48:37.084092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.084086    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a73aaab0dc127afef693d22b535b7e180f95dadc3cbab4233c53a334c45171be"
Oct 30 21:48:37.235039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.235002    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused" start-of-body=
Oct 30 21:48:37.235208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.235052    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.120:15021/healthz/ready\": dial tcp 10.132.1.120:15021: connect: connection refused"
Oct 30 21:48:37.442577 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.442552    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:48:37.556199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.543023    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-podinfo\") pod \"431ab330-ad11-4272-9adf-1584a08e3aaf\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") "
Oct 30 21:48:37.556199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.543117    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-envoy\") pod \"431ab330-ad11-4272-9adf-1584a08e3aaf\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") "
Oct 30 21:48:37.556199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.543222    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-kube-api-access-j5wf4\" (UniqueName: \"kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-kube-api-access-j5wf4\") pod \"431ab330-ad11-4272-9adf-1584a08e3aaf\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") "
Oct 30 21:48:37.556199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.543273    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-credential-socket\") pod \"431ab330-ad11-4272-9adf-1584a08e3aaf\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") "
Oct 30 21:48:37.556199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.543381    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-data\") pod \"431ab330-ad11-4272-9adf-1584a08e3aaf\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") "
Oct 30 21:48:37.556199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.543437    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-socket\") pod \"431ab330-ad11-4272-9adf-1584a08e3aaf\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") "
Oct 30 21:48:37.556199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.543547    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-certs\") pod \"431ab330-ad11-4272-9adf-1584a08e3aaf\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") "
Oct 30 21:48:37.556199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.543599    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-token\" (UniqueName: \"kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-token\") pod \"431ab330-ad11-4272-9adf-1584a08e3aaf\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") "
Oct 30 21:48:37.556199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.543634    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/431ab330-ad11-4272-9adf-1584a08e3aaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431ab330-ad11-4272-9adf-1584a08e3aaf-istiod-ca-cert\") pod \"431ab330-ad11-4272-9adf-1584a08e3aaf\" (UID: \"431ab330-ad11-4272-9adf-1584a08e3aaf\") "
Oct 30 21:48:37.556199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.555714    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-podinfo" pod "431ab330-ad11-4272-9adf-1584a08e3aaf" (UID: "431ab330-ad11-4272-9adf-1584a08e3aaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:48:37.562065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.559621    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/431ab330-ad11-4272-9adf-1584a08e3aaf-istiod-ca-cert" pod "431ab330-ad11-4272-9adf-1584a08e3aaf" (UID: "431ab330-ad11-4272-9adf-1584a08e3aaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:37.562065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.560017    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-data" pod "431ab330-ad11-4272-9adf-1584a08e3aaf" (UID: "431ab330-ad11-4272-9adf-1584a08e3aaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:37.562923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.562748    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-certs" pod "431ab330-ad11-4272-9adf-1584a08e3aaf" (UID: "431ab330-ad11-4272-9adf-1584a08e3aaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:37.563196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.563108    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-socket" pod "431ab330-ad11-4272-9adf-1584a08e3aaf" (UID: "431ab330-ad11-4272-9adf-1584a08e3aaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:37.563406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.563366    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-credential-socket" pod "431ab330-ad11-4272-9adf-1584a08e3aaf" (UID: "431ab330-ad11-4272-9adf-1584a08e3aaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:37.572653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.572611    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-envoy" pod "431ab330-ad11-4272-9adf-1584a08e3aaf" (UID: "431ab330-ad11-4272-9adf-1584a08e3aaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:37.575290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.575257    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-kube-api-access-j5wf4" pod "431ab330-ad11-4272-9adf-1584a08e3aaf" (UID: "431ab330-ad11-4272-9adf-1584a08e3aaf"). InnerVolumeSpecName "kube-api-access-j5wf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:37.575849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.575815    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-token" pod "431ab330-ad11-4272-9adf-1584a08e3aaf" (UID: "431ab330-ad11-4272-9adf-1584a08e3aaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:37.646721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.646631    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.646721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.646667    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.646721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.646684    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.646721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.646701    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.646721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.646715    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/431ab330-ad11-4272-9adf-1584a08e3aaf-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.647072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.646731    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.647072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.646745    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.647072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.646761    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5wf4\" (UniqueName: \"kubernetes.io/projected/431ab330-ad11-4272-9adf-1584a08e3aaf-kube-api-access-j5wf4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:37.647072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:37.646776    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/431ab330-ad11-4272-9adf-1584a08e3aaf-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:38.094521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:38.094485    4918 generic.go:356] "Generic (PLEG): container finished" podID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerID="874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8" exitCode=0
Oct 30 21:48:38.094684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:38.094528    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" event={"ID":"431ab330-ad11-4272-9adf-1584a08e3aaf","Type":"ContainerDied","Data":"874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8"}
Oct 30 21:48:38.094684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:38.094543    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"
Oct 30 21:48:38.094684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:38.094577    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl" event={"ID":"431ab330-ad11-4272-9adf-1584a08e3aaf","Type":"ContainerDied","Data":"42433ff58148a98acec40c89b8dbe1d1b4e8b6033f1db26931760b965a594fb9"}
Oct 30 21:48:38.094684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:38.094606    4918 scope.go:122] "RemoveContainer" containerID="874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8"
Oct 30 21:48:38.107946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:38.107930    4918 scope.go:122] "RemoveContainer" containerID="874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8"
Oct 30 21:48:38.108273 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:48:38.108249    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8\": container with ID starting with 874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8 not found: ID does not exist" containerID="874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8"
Oct 30 21:48:38.108367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:38.108288    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8"} err="failed to get container status \"874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8\": rpc error: code = NotFound desc = could not find container \"874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8\": container with ID starting with 874ac123d15a4457697c3b34aebf7a95ceb3f1f52f1bb9d56f778e1a36eda4b8 not found: ID does not exist"
Oct 30 21:48:38.126698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:38.126672    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"]
Oct 30 21:48:38.139317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:38.139294    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b30a42-openshift-default-664cb6454c-xwdkl"]
Oct 30 21:48:39.081018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.080984    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" path="/var/lib/kubelet/pods/431ab330-ad11-4272-9adf-1584a08e3aaf/volumes"
Oct 30 21:48:39.171284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.171251    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"]
Oct 30 21:48:39.172357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.172333    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9581fcef-e655-4006-8561-7643c3852e01" containerName="revoke-keys"
Oct 30 21:48:39.172357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.172355    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9581fcef-e655-4006-8561-7643c3852e01" containerName="revoke-keys"
Oct 30 21:48:39.172584 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.172373    4918 cpu_manager.go:414] "containerMap: removing container" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy"
Oct 30 21:48:39.172584 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.172378    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy"
Oct 30 21:48:39.172584 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.172581    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9581fcef-e655-4006-8561-7643c3852e01" containerName="revoke-keys"
Oct 30 21:48:39.172770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.172591    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="431ab330-ad11-4272-9adf-1584a08e3aaf" containerName="istio-proxy"
Oct 30 21:48:39.179165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.179146    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.187342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.187313    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:48:39.198887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.198708    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rx79k\""
Oct 30 21:48:39.198887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.198786    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-001667-serving-cert\""
Oct 30 21:48:39.199752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.199731    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"]
Oct 30 21:48:39.201415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.200696    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:48:39.206994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.203828    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:48:39.276551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.276519    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7ce593-4a73-498b-b239-67d29a348efe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-001667-c9f875749-xc4mw\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.276759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.276744    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-maas-api-tls\") pod \"maas-api-e2e-inf-b-001667-c9f875749-xc4mw\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.276884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.276867    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-metrics-certs\") pod \"maas-api-e2e-inf-b-001667-c9f875749-xc4mw\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.277043 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.277025    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmpc2\" (UniqueName: \"kubernetes.io/projected/2a7ce593-4a73-498b-b239-67d29a348efe-kube-api-access-gmpc2\") pod \"maas-api-e2e-inf-b-001667-c9f875749-xc4mw\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.404474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.396163    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-metrics-certs\") pod \"maas-api-e2e-inf-b-001667-c9f875749-xc4mw\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.404474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.396377    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmpc2\" (UniqueName: \"kubernetes.io/projected/2a7ce593-4a73-498b-b239-67d29a348efe-kube-api-access-gmpc2\") pod \"maas-api-e2e-inf-b-001667-c9f875749-xc4mw\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.404474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.396547    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7ce593-4a73-498b-b239-67d29a348efe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-001667-c9f875749-xc4mw\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.404474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.396720    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-maas-api-tls\") pod \"maas-api-e2e-inf-b-001667-c9f875749-xc4mw\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.414424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.405103    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7ce593-4a73-498b-b239-67d29a348efe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-001667-c9f875749-xc4mw\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.442338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.431594    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-metrics-certs\") pod \"maas-api-e2e-inf-b-001667-c9f875749-xc4mw\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.460414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.453270    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmpc2\" (UniqueName: \"kubernetes.io/projected/2a7ce593-4a73-498b-b239-67d29a348efe-kube-api-access-gmpc2\") pod \"maas-api-e2e-inf-b-001667-c9f875749-xc4mw\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.467559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.466979    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-maas-api-tls\") pod \"maas-api-e2e-inf-b-001667-c9f875749-xc4mw\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.514106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.514075    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:39.995411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:39.991875    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"]
Oct 30 21:48:40.009608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:40.008726    4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:48:40.123666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:40.123626    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw" event={"ID":"2a7ce593-4a73-498b-b239-67d29a348efe","Type":"ContainerStarted","Data":"f45370f9a165b7edb889a578bc7a51b16f7b6a856a7b90ef848a41b79628b7e2"}
Oct 30 21:48:42.798132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:42.798099    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"]
Oct 30 21:48:42.821126 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:42.821079    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"]
Oct 30 21:48:45.336111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.336053    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw" event={"ID":"2a7ce593-4a73-498b-b239-67d29a348efe","Type":"ContainerStarted","Data":"afd3c89b6d188d438a5b5cbcb5b8642c63c261f065a6a7fccd9e7e9bcab7890f"}
Oct 30 21:48:45.336937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.336368    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:45.361719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.361664    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw" podStartSLOduration=1.746385304 podStartE2EDuration="6.361650137s" totalImagesPullingTime="4.615264833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:45.360548947 +0000 UTC m=+2838.407108091" watchObservedRunningTime="2026-10-30 21:48:45.361650137 +0000 UTC m=+2838.408209282"
Oct 30 21:48:45.715507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.715433    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"]
Oct 30 21:48:45.720327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.720307    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:45.730120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.730099    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:48:45.730284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.730267    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:48:45.745746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.745722    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"]
Oct 30 21:48:45.847590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.847556    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7bb9d6-9btcd\" (UID: \"093b41dd-3056-45ce-a515-c7d3fb315992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:45.847745 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.847698    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zvr8\" (UniqueName: \"kubernetes.io/projected/093b41dd-3056-45ce-a515-c7d3fb315992-kube-api-access-5zvr8\") pod \"maas-api-revoke-sub-9c7bb9d6-9btcd\" (UID: \"093b41dd-3056-45ce-a515-c7d3fb315992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:45.847916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.847900    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7bb9d6-9btcd\" (UID: \"093b41dd-3056-45ce-a515-c7d3fb315992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:45.949327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.949292    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7bb9d6-9btcd\" (UID: \"093b41dd-3056-45ce-a515-c7d3fb315992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:45.949547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.949528    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7bb9d6-9btcd\" (UID: \"093b41dd-3056-45ce-a515-c7d3fb315992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:45.949660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.949644    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvr8\" (UniqueName: \"kubernetes.io/projected/093b41dd-3056-45ce-a515-c7d3fb315992-kube-api-access-5zvr8\") pod \"maas-api-revoke-sub-9c7bb9d6-9btcd\" (UID: \"093b41dd-3056-45ce-a515-c7d3fb315992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:45.950202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.950181    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7bb9d6-9btcd\" (UID: \"093b41dd-3056-45ce-a515-c7d3fb315992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:45.951880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.951864    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7bb9d6-9btcd\" (UID: \"093b41dd-3056-45ce-a515-c7d3fb315992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:45.965123 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:45.965097    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zvr8\" (UniqueName: \"kubernetes.io/projected/093b41dd-3056-45ce-a515-c7d3fb315992-kube-api-access-5zvr8\") pod \"maas-api-revoke-sub-9c7bb9d6-9btcd\" (UID: \"093b41dd-3056-45ce-a515-c7d3fb315992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:46.036181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:46.036144    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:46.331224 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:48:46.331192    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093b41dd_3056_45ce_a515_c7d3fb315992.slice/crio-b27cfc768ce6e4b4cd8d04fd91177c3b87951405bf3241e555c37a488c560c03 WatchSource:0}: Error finding container b27cfc768ce6e4b4cd8d04fd91177c3b87951405bf3241e555c37a488c560c03: Status 404 returned error can't find the container with id b27cfc768ce6e4b4cd8d04fd91177c3b87951405bf3241e555c37a488c560c03
Oct 30 21:48:46.333298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:46.333275    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"]
Oct 30 21:48:46.346314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:46.346289    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd" event={"ID":"093b41dd-3056-45ce-a515-c7d3fb315992","Type":"ContainerStarted","Data":"b27cfc768ce6e4b4cd8d04fd91177c3b87951405bf3241e555c37a488c560c03"}
Oct 30 21:48:47.439416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:47.431791    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd" event={"ID":"093b41dd-3056-45ce-a515-c7d3fb315992","Type":"ContainerStarted","Data":"2cc5c9ad33d7a5c2e85890354b544d0dca4a2b7cee179e04a12add93c5cb0d21"}
Oct 30 21:48:47.605598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:47.593116    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd" podStartSLOduration=2.5930927329999998 podStartE2EDuration="2.593092733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:47.515476048 +0000 UTC m=+2840.562035194" watchObservedRunningTime="2026-10-30 21:48:47.593092733 +0000 UTC m=+2840.639651878"
Oct 30 21:48:47.799217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:47.799175    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6" podUID="0256d9fa-a55f-4457-baf2-b337961bcadd" containerName="payload-processing" containerID="cri-o://74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de" gracePeriod=25
Oct 30 21:48:47.821967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:47.821928    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f" podUID="53f737dc-6ea7-4281-a418-7b84fcf2fa4c" containerName="payload-pre-processing" containerID="cri-o://81f586fa9e61d5bc360541cb79d4d14536243ab957ccc85a208d49675fed98bc" gracePeriod=25
Oct 30 21:48:48.442524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:48.442471    4918 generic.go:356] "Generic (PLEG): container finished" podID="093b41dd-3056-45ce-a515-c7d3fb315992" containerID="2cc5c9ad33d7a5c2e85890354b544d0dca4a2b7cee179e04a12add93c5cb0d21" exitCode=0
Oct 30 21:48:48.442956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:48.442573    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd" event={"ID":"093b41dd-3056-45ce-a515-c7d3fb315992","Type":"ContainerDied","Data":"2cc5c9ad33d7a5c2e85890354b544d0dca4a2b7cee179e04a12add93c5cb0d21"}
Oct 30 21:48:50.026480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.026370    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:50.146473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.136542    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-xks-ca\") pod \"093b41dd-3056-45ce-a515-c7d3fb315992\" (UID: \"093b41dd-3056-45ce-a515-c7d3fb315992\") "
Oct 30 21:48:50.146473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.136766    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093b41dd-3056-45ce-a515-c7d3fb315992-kube-api-access-5zvr8\" (UniqueName: \"kubernetes.io/projected/093b41dd-3056-45ce-a515-c7d3fb315992-kube-api-access-5zvr8\") pod \"093b41dd-3056-45ce-a515-c7d3fb315992\" (UID: \"093b41dd-3056-45ce-a515-c7d3fb315992\") "
Oct 30 21:48:50.146473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.136912    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-openshift-ca\") pod \"093b41dd-3056-45ce-a515-c7d3fb315992\" (UID: \"093b41dd-3056-45ce-a515-c7d3fb315992\") "
Oct 30 21:48:50.169207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.167983    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-openshift-ca" pod "093b41dd-3056-45ce-a515-c7d3fb315992" (UID: "093b41dd-3056-45ce-a515-c7d3fb315992"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:50.169207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.169037    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:50.169724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.169700    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093b41dd-3056-45ce-a515-c7d3fb315992-kube-api-access-5zvr8" pod "093b41dd-3056-45ce-a515-c7d3fb315992" (UID: "093b41dd-3056-45ce-a515-c7d3fb315992"). InnerVolumeSpecName "kube-api-access-5zvr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:50.272511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.272479    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-xks-ca" pod "093b41dd-3056-45ce-a515-c7d3fb315992" (UID: "093b41dd-3056-45ce-a515-c7d3fb315992"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:50.272785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.272608    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zvr8\" (UniqueName: \"kubernetes.io/projected/093b41dd-3056-45ce-a515-c7d3fb315992-kube-api-access-5zvr8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:50.374583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.374552    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093b41dd-3056-45ce-a515-c7d3fb315992-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:50.462141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.462106    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"
Oct 30 21:48:50.462141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.462135    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd" event={"ID":"093b41dd-3056-45ce-a515-c7d3fb315992","Type":"ContainerDied","Data":"b27cfc768ce6e4b4cd8d04fd91177c3b87951405bf3241e555c37a488c560c03"}
Oct 30 21:48:50.462353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:50.462186    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b27cfc768ce6e4b4cd8d04fd91177c3b87951405bf3241e555c37a488c560c03"
Oct 30 21:48:51.392618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:51.392585    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:48:51.561845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:51.561805    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"]
Oct 30 21:48:51.567750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:51.567724    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-128d2f-ghdp8"]
Oct 30 21:48:53.082259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:53.082226    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9581fcef-e655-4006-8561-7643c3852e01" path="/var/lib/kubelet/pods/9581fcef-e655-4006-8561-7643c3852e01/volumes"
Oct 30 21:48:53.528291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:53.528253    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6" podUID="0256d9fa-a55f-4457-baf2-b337961bcadd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:48:54.588187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:54.588086    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f" podUID="53f737dc-6ea7-4281-a418-7b84fcf2fa4c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:48:56.233582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.233552    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"]
Oct 30 21:48:56.234489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.234472    4918 cpu_manager.go:414] "containerMap: removing container" podUID="093b41dd-3056-45ce-a515-c7d3fb315992" containerName="revoke-keys"
Oct 30 21:48:56.234489 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.234489    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093b41dd-3056-45ce-a515-c7d3fb315992" containerName="revoke-keys"
Oct 30 21:48:56.234680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.234671    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="093b41dd-3056-45ce-a515-c7d3fb315992" containerName="revoke-keys"
Oct 30 21:48:56.238744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.238723    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:48:56.248040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.246327    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:48:56.248040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.247080    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:48:56.248469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.248447    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"]
Oct 30 21:48:56.383806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.383760    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5c3268-t5xnz\" (UID: \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:48:56.384073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.384049    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5c3268-t5xnz\" (UID: \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:48:56.384180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.384158    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cdkq\" (UniqueName: \"kubernetes.io/projected/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-kube-api-access-8cdkq\") pod \"maas-api-revoke-sub-0b5c3268-t5xnz\" (UID: \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:48:56.485710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.485625    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cdkq\" (UniqueName: \"kubernetes.io/projected/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-kube-api-access-8cdkq\") pod \"maas-api-revoke-sub-0b5c3268-t5xnz\" (UID: \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:48:56.485902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.485798    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5c3268-t5xnz\" (UID: \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:48:56.486079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.486061    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5c3268-t5xnz\" (UID: \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:48:56.486458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.486439    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5c3268-t5xnz\" (UID: \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:48:56.488832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.488807    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5c3268-t5xnz\" (UID: \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:48:56.500264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.500234    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cdkq\" (UniqueName: \"kubernetes.io/projected/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-kube-api-access-8cdkq\") pod \"maas-api-revoke-sub-0b5c3268-t5xnz\" (UID: \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:48:56.559183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.559149    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:48:56.958263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:56.958235    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"]
Oct 30 21:48:56.958824 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:48:56.958793    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a09178f_5a3a_4437_92fb_73f0e4a5ace0.slice/crio-cb1e125d57af78b6b6e7dab731a17cbaa2361120982a6fa92f5a29432adcd517 WatchSource:0}: Error finding container cb1e125d57af78b6b6e7dab731a17cbaa2361120982a6fa92f5a29432adcd517: Status 404 returned error can't find the container with id cb1e125d57af78b6b6e7dab731a17cbaa2361120982a6fa92f5a29432adcd517
Oct 30 21:48:57.754268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:57.754225    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz" event={"ID":"7a09178f-5a3a-4437-92fb-73f0e4a5ace0","Type":"ContainerStarted","Data":"97be1ef4748988ec0b65fce340430d412cbe17beeef83618037c04fff971de38"}
Oct 30 21:48:57.755150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:57.755122    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz" event={"ID":"7a09178f-5a3a-4437-92fb-73f0e4a5ace0","Type":"ContainerStarted","Data":"cb1e125d57af78b6b6e7dab731a17cbaa2361120982a6fa92f5a29432adcd517"}
Oct 30 21:48:57.787656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:57.787605    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz" podStartSLOduration=1.787589705 podStartE2EDuration="1.787589705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:57.783162236 +0000 UTC m=+2850.829721383" watchObservedRunningTime="2026-10-30 21:48:57.787589705 +0000 UTC m=+2850.834148884"
Oct 30 21:48:58.764407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:58.764348    4918 generic.go:356] "Generic (PLEG): container finished" podID="7a09178f-5a3a-4437-92fb-73f0e4a5ace0" containerID="97be1ef4748988ec0b65fce340430d412cbe17beeef83618037c04fff971de38" exitCode=0
Oct 30 21:48:58.764880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:58.764437    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz" event={"ID":"7a09178f-5a3a-4437-92fb-73f0e4a5ace0","Type":"ContainerDied","Data":"97be1ef4748988ec0b65fce340430d412cbe17beeef83618037c04fff971de38"}
Oct 30 21:48:58.766487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:58.766456    4918 generic.go:356] "Generic (PLEG): container finished" podID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerID="acf0d574c3e6a38883f1b3712e0dd3bd6c8d5a308f896a051e7068e28d4ada50" exitCode=0
Oct 30 21:48:58.766487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:58.766469    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" event={"ID":"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59","Type":"ContainerDied","Data":"acf0d574c3e6a38883f1b3712e0dd3bd6c8d5a308f896a051e7068e28d4ada50"}
Oct 30 21:48:58.766917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:58.766904    4918 scope.go:122] "RemoveContainer" containerID="acf0d574c3e6a38883f1b3712e0dd3bd6c8d5a308f896a051e7068e28d4ada50"
Oct 30 21:48:58.768930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:58.768905    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:48:58.768997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:58.768972    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:48:58.769040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:48:58.769006    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:48:59.166649 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:48:59.166555    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:49:00.365818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.365768    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" event={"ID":"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59","Type":"ContainerStarted","Data":"84d9f022521d2ef3b21f4d4119840f2fc525110c4b365619215477ed1c1d3be2"}
Oct 30 21:49:00.622772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.622676    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:49:00.623805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.623767    4918 patch_prober.go:29] interesting pod/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 21:49:00.623937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.623830    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 21:49:00.769415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.763588    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:49:00.863500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.861612    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-kube-api-access-8cdkq\" (UniqueName: \"kubernetes.io/projected/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-kube-api-access-8cdkq\") pod \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\" (UID: \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\") "
Oct 30 21:49:00.863500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.861946    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-openshift-ca\") pod \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\" (UID: \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\") "
Oct 30 21:49:00.863500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.862098    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-xks-ca\") pod \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\" (UID: \"7a09178f-5a3a-4437-92fb-73f0e4a5ace0\") "
Oct 30 21:49:00.864106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.864050    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-openshift-ca" pod "7a09178f-5a3a-4437-92fb-73f0e4a5ace0" (UID: "7a09178f-5a3a-4437-92fb-73f0e4a5ace0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:00.883327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.883230    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-kube-api-access-8cdkq" pod "7a09178f-5a3a-4437-92fb-73f0e4a5ace0" (UID: "7a09178f-5a3a-4437-92fb-73f0e4a5ace0"). InnerVolumeSpecName "kube-api-access-8cdkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:00.899435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.895339    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-xks-ca" pod "7a09178f-5a3a-4437-92fb-73f0e4a5ace0" (UID: "7a09178f-5a3a-4437-92fb-73f0e4a5ace0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:00.964357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.964326    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:00.964357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.964355    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:00.964588 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:00.964367    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cdkq\" (UniqueName: \"kubernetes.io/projected/7a09178f-5a3a-4437-92fb-73f0e4a5ace0-kube-api-access-8cdkq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:01.376138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:01.376107    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"
Oct 30 21:49:01.376583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:01.376135    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz" event={"ID":"7a09178f-5a3a-4437-92fb-73f0e4a5ace0","Type":"ContainerDied","Data":"cb1e125d57af78b6b6e7dab731a17cbaa2361120982a6fa92f5a29432adcd517"}
Oct 30 21:49:01.376583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:01.376200    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb1e125d57af78b6b6e7dab731a17cbaa2361120982a6fa92f5a29432adcd517"
Oct 30 21:49:01.513836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:01.513795    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:49:01.592426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:01.591771    4918 patch_prober.go:29] interesting pod/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 21:49:01.592426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:01.591833    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 21:49:02.610955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:02.610839    4918 patch_prober.go:29] interesting pod/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 21:49:02.610955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:02.610897    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 21:49:03.528523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:03.528479    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6" podUID="0256d9fa-a55f-4457-baf2-b337961bcadd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:49:03.608415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:03.607863    4918 patch_prober.go:29] interesting pod/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 21:49:03.608415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:03.607927    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 21:49:04.606415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:04.603006    4918 patch_prober.go:29] interesting pod/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 21:49:04.606415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:04.603068    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 21:49:04.611720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:04.611433    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f" podUID="53f737dc-6ea7-4281-a418-7b84fcf2fa4c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:49:05.321443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:05.321285    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"]
Oct 30 21:49:05.327868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:05.327837    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"]
Oct 30 21:49:05.606470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:05.605836    4918 patch_prober.go:29] interesting pod/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused" start-of-body=
Oct 30 21:49:05.606470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:05.605900    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.136:15021/healthz/ready\": dial tcp 10.132.1.136:15021: connect: connection refused"
Oct 30 21:49:06.593324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:06.593298    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:49:06.594451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:06.594426    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:49:08.393482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.380297    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"
Oct 30 21:49:08.628412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.520228    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0256d9fa-a55f-4457-baf2-b337961bcadd-kube-api-access-qsrfv\" (UniqueName: \"kubernetes.io/projected/0256d9fa-a55f-4457-baf2-b337961bcadd-kube-api-access-qsrfv\") pod \"0256d9fa-a55f-4457-baf2-b337961bcadd\" (UID: \"0256d9fa-a55f-4457-baf2-b337961bcadd\") "
Oct 30 21:49:08.628412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.520804    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0256d9fa-a55f-4457-baf2-b337961bcadd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0256d9fa-a55f-4457-baf2-b337961bcadd-plugins-config-volume\") pod \"0256d9fa-a55f-4457-baf2-b337961bcadd\" (UID: \"0256d9fa-a55f-4457-baf2-b337961bcadd\") "
Oct 30 21:49:08.713201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.713110    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0256d9fa-a55f-4457-baf2-b337961bcadd-plugins-config-volume" pod "0256d9fa-a55f-4457-baf2-b337961bcadd" (UID: "0256d9fa-a55f-4457-baf2-b337961bcadd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:08.787426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.780238    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0256d9fa-a55f-4457-baf2-b337961bcadd-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:08.949785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.948483    4918 generic.go:356] "Generic (PLEG): container finished" podID="0256d9fa-a55f-4457-baf2-b337961bcadd" containerID="74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de" exitCode=0
Oct 30 21:49:08.949785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.948814    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6" event={"ID":"0256d9fa-a55f-4457-baf2-b337961bcadd","Type":"ContainerDied","Data":"74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de"}
Oct 30 21:49:08.949785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.948889    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6" event={"ID":"0256d9fa-a55f-4457-baf2-b337961bcadd","Type":"ContainerDied","Data":"cea50ec0a6bf7a4c685af591f940bc8d7442fe85bab556d181ffcda66ee4c863"}
Oct 30 21:49:08.949785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.948938    4918 scope.go:122] "RemoveContainer" containerID="74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de"
Oct 30 21:49:08.949785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.949196    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"
Oct 30 21:49:08.964941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.960578    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0256d9fa-a55f-4457-baf2-b337961bcadd-kube-api-access-qsrfv" pod "0256d9fa-a55f-4457-baf2-b337961bcadd" (UID: "0256d9fa-a55f-4457-baf2-b337961bcadd"). InnerVolumeSpecName "kube-api-access-qsrfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:08.980364 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.980332    4918 generic.go:356] "Generic (PLEG): container finished" podID="53f737dc-6ea7-4281-a418-7b84fcf2fa4c" containerID="81f586fa9e61d5bc360541cb79d4d14536243ab957ccc85a208d49675fed98bc" exitCode=0
Oct 30 21:49:08.980863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:08.980837    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f" event={"ID":"53f737dc-6ea7-4281-a418-7b84fcf2fa4c","Type":"ContainerDied","Data":"81f586fa9e61d5bc360541cb79d4d14536243ab957ccc85a208d49675fed98bc"}
Oct 30 21:49:09.019413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.013623    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsrfv\" (UniqueName: \"kubernetes.io/projected/0256d9fa-a55f-4457-baf2-b337961bcadd-kube-api-access-qsrfv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:09.306294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.306208    4918 scope.go:122] "RemoveContainer" containerID="74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de"
Oct 30 21:49:09.307509 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:49:09.307480    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de\": container with ID starting with 74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de not found: ID does not exist" containerID="74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de"
Oct 30 21:49:09.307732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.307705    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de"} err="failed to get container status \"74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de\": rpc error: code = NotFound desc = could not find container \"74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de\": container with ID starting with 74e29bb1201f007725a055e5f80a1f865133b900a68d06d54c456b445123e6de not found: ID does not exist"
Oct 30 21:49:09.504427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.500869    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"
Oct 30 21:49:09.700472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.638134    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"]
Oct 30 21:49:09.700472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.649607    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-plugins-config-volume\") pod \"53f737dc-6ea7-4281-a418-7b84fcf2fa4c\" (UID: \"53f737dc-6ea7-4281-a418-7b84fcf2fa4c\") "
Oct 30 21:49:09.700472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.652688    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-plugins-config-volume" pod "53f737dc-6ea7-4281-a418-7b84fcf2fa4c" (UID: "53f737dc-6ea7-4281-a418-7b84fcf2fa4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:09.700472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.680185    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-kube-api-access-6glxg\" (UniqueName: \"kubernetes.io/projected/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-kube-api-access-6glxg\") pod \"53f737dc-6ea7-4281-a418-7b84fcf2fa4c\" (UID: \"53f737dc-6ea7-4281-a418-7b84fcf2fa4c\") "
Oct 30 21:49:09.700472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.688513    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:09.700472 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.693858    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-kube-api-access-6glxg" pod "53f737dc-6ea7-4281-a418-7b84fcf2fa4c" (UID: "53f737dc-6ea7-4281-a418-7b84fcf2fa4c"). InnerVolumeSpecName "kube-api-access-6glxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:09.753816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.747425    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-128d2f-74c8dbc6d9-45ph6"]
Oct 30 21:49:09.790614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.790583    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6glxg\" (UniqueName: \"kubernetes.io/projected/53f737dc-6ea7-4281-a418-7b84fcf2fa4c-kube-api-access-6glxg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:09.994759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.994726    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"
Oct 30 21:49:09.994965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.994940    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f" event={"ID":"53f737dc-6ea7-4281-a418-7b84fcf2fa4c","Type":"ContainerDied","Data":"4730653bc9794868504ec0cb39a946fcda7392ac85786889b22aa0808fc10126"}
Oct 30 21:49:09.995128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:09.995113    4918 scope.go:122] "RemoveContainer" containerID="81f586fa9e61d5bc360541cb79d4d14536243ab957ccc85a208d49675fed98bc"
Oct 30 21:49:10.034277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:10.034246    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"]
Oct 30 21:49:10.051470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:10.051433    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-128d2f-7465f8f5ff-2p48f"]
Oct 30 21:49:10.772066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:10.772024    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"]
Oct 30 21:49:11.700415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:11.700269    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0256d9fa-a55f-4457-baf2-b337961bcadd" path="/var/lib/kubelet/pods/0256d9fa-a55f-4457-baf2-b337961bcadd/volumes"
Oct 30 21:49:11.726415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:11.701309    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f737dc-6ea7-4281-a418-7b84fcf2fa4c" path="/var/lib/kubelet/pods/53f737dc-6ea7-4281-a418-7b84fcf2fa4c/volumes"
Oct 30 21:49:15.773135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:15.773069    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g" podUID="036bea86-70eb-48e1-b108-b71928175a68" containerName="payload-processing" containerID="cri-o://799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb" gracePeriod=25
Oct 30 21:49:17.436962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.436919    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"]
Oct 30 21:49:17.437839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.437823    4918 cpu_manager.go:414] "containerMap: removing container" podUID="0256d9fa-a55f-4457-baf2-b337961bcadd" containerName="payload-processing"
Oct 30 21:49:17.437839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.437839    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0256d9fa-a55f-4457-baf2-b337961bcadd" containerName="payload-processing"
Oct 30 21:49:17.437952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.437887    4918 cpu_manager.go:414] "containerMap: removing container" podUID="53f737dc-6ea7-4281-a418-7b84fcf2fa4c" containerName="payload-pre-processing"
Oct 30 21:49:17.437952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.437893    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f737dc-6ea7-4281-a418-7b84fcf2fa4c" containerName="payload-pre-processing"
Oct 30 21:49:17.437952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.437905    4918 cpu_manager.go:414] "containerMap: removing container" podUID="7a09178f-5a3a-4437-92fb-73f0e4a5ace0" containerName="revoke-keys"
Oct 30 21:49:17.437952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.437911    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a09178f-5a3a-4437-92fb-73f0e4a5ace0" containerName="revoke-keys"
Oct 30 21:49:17.438089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.438048    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f737dc-6ea7-4281-a418-7b84fcf2fa4c" containerName="payload-pre-processing"
Oct 30 21:49:17.438089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.438056    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a09178f-5a3a-4437-92fb-73f0e4a5ace0" containerName="revoke-keys"
Oct 30 21:49:17.438089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.438067    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0256d9fa-a55f-4457-baf2-b337961bcadd" containerName="payload-processing"
Oct 30 21:49:17.444015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.443994    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:17.449738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.449716    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:49:17.450072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.450054    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:49:17.456106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.456076    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"]
Oct 30 21:49:17.462732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.462710    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"]
Oct 30 21:49:17.462844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.462791    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:17.483002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.482971    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"]
Oct 30 21:49:17.577991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.577956    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69611331-493d-459d-83b1-e977506590af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2d559a-trwcb\" (UID: \"69611331-493d-459d-83b1-e977506590af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:17.578178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.578118    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0a4b240-htsl2\" (UID: \"0138ea77-3423-43be-abbb-404ea9e94c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:17.578178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.578155    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pfkz\" (UniqueName: \"kubernetes.io/projected/0138ea77-3423-43be-abbb-404ea9e94c8c-kube-api-access-7pfkz\") pod \"maas-api-revoke-sub-d0a4b240-htsl2\" (UID: \"0138ea77-3423-43be-abbb-404ea9e94c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:17.578178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.578174    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0a4b240-htsl2\" (UID: \"0138ea77-3423-43be-abbb-404ea9e94c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:17.578411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.578369    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfsjb\" (UniqueName: \"kubernetes.io/projected/69611331-493d-459d-83b1-e977506590af-kube-api-access-rfsjb\") pod \"maas-api-revoke-sub-5c2d559a-trwcb\" (UID: \"69611331-493d-459d-83b1-e977506590af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:17.578492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.578447    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69611331-493d-459d-83b1-e977506590af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2d559a-trwcb\" (UID: \"69611331-493d-459d-83b1-e977506590af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:17.679630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.679597    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0a4b240-htsl2\" (UID: \"0138ea77-3423-43be-abbb-404ea9e94c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:17.679775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.679669    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pfkz\" (UniqueName: \"kubernetes.io/projected/0138ea77-3423-43be-abbb-404ea9e94c8c-kube-api-access-7pfkz\") pod \"maas-api-revoke-sub-d0a4b240-htsl2\" (UID: \"0138ea77-3423-43be-abbb-404ea9e94c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:17.679775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.679705    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0a4b240-htsl2\" (UID: \"0138ea77-3423-43be-abbb-404ea9e94c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:17.679923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.679906    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfsjb\" (UniqueName: \"kubernetes.io/projected/69611331-493d-459d-83b1-e977506590af-kube-api-access-rfsjb\") pod \"maas-api-revoke-sub-5c2d559a-trwcb\" (UID: \"69611331-493d-459d-83b1-e977506590af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:17.679990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.679949    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69611331-493d-459d-83b1-e977506590af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2d559a-trwcb\" (UID: \"69611331-493d-459d-83b1-e977506590af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:17.680091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.680046    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69611331-493d-459d-83b1-e977506590af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2d559a-trwcb\" (UID: \"69611331-493d-459d-83b1-e977506590af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:17.680512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.680489    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0a4b240-htsl2\" (UID: \"0138ea77-3423-43be-abbb-404ea9e94c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:17.680750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.680727    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69611331-493d-459d-83b1-e977506590af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c2d559a-trwcb\" (UID: \"69611331-493d-459d-83b1-e977506590af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:17.682322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.682300    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0a4b240-htsl2\" (UID: \"0138ea77-3423-43be-abbb-404ea9e94c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:17.682769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.682749    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69611331-493d-459d-83b1-e977506590af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c2d559a-trwcb\" (UID: \"69611331-493d-459d-83b1-e977506590af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:17.696949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.696895    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pfkz\" (UniqueName: \"kubernetes.io/projected/0138ea77-3423-43be-abbb-404ea9e94c8c-kube-api-access-7pfkz\") pod \"maas-api-revoke-sub-d0a4b240-htsl2\" (UID: \"0138ea77-3423-43be-abbb-404ea9e94c8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:17.701598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.701575    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfsjb\" (UniqueName: \"kubernetes.io/projected/69611331-493d-459d-83b1-e977506590af-kube-api-access-rfsjb\") pod \"maas-api-revoke-sub-5c2d559a-trwcb\" (UID: \"69611331-493d-459d-83b1-e977506590af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:17.763725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.763694    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:17.790134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:17.790104    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:18.047718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:18.047651    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"]
Oct 30 21:49:18.059684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:18.059632    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72cedf6b-bcszs"]
Oct 30 21:49:18.079438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:18.079151    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"]
Oct 30 21:49:18.098493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:18.098465    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"]
Oct 30 21:49:18.106603 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:49:18.106574    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0138ea77_3423_43be_abbb_404ea9e94c8c.slice/crio-c0a3d06417ca5152576d38158e19af21e0349357fbc392b36ad461e825275da4 WatchSource:0}: Error finding container c0a3d06417ca5152576d38158e19af21e0349357fbc392b36ad461e825275da4: Status 404 returned error can't find the container with id c0a3d06417ca5152576d38158e19af21e0349357fbc392b36ad461e825275da4
Oct 30 21:49:18.131409 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:18.131362    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2" event={"ID":"0138ea77-3423-43be-abbb-404ea9e94c8c","Type":"ContainerStarted","Data":"c0a3d06417ca5152576d38158e19af21e0349357fbc392b36ad461e825275da4"}
Oct 30 21:49:18.133074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:18.133041    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb" event={"ID":"69611331-493d-459d-83b1-e977506590af","Type":"ContainerStarted","Data":"5850bc6333bd174f94f16bfbcc8bbd857384a13783821cc3b7103fe9a9b74f7b"}
Oct 30 21:49:19.080658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:19.080627    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e25da316-885a-432d-b00c-fa70d628c57c" path="/var/lib/kubelet/pods/e25da316-885a-432d-b00c-fa70d628c57c/volumes"
Oct 30 21:49:19.143808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:19.143775    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb" event={"ID":"69611331-493d-459d-83b1-e977506590af","Type":"ContainerStarted","Data":"28c81b41f0be9d7a76f06fea787f7ec8014748a84f63cb1db4e50e2936c0cb76"}
Oct 30 21:49:19.145791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:19.145762    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2" event={"ID":"0138ea77-3423-43be-abbb-404ea9e94c8c","Type":"ContainerStarted","Data":"0032c21601142c952da06f52618680406b9f93f123defcac8bedc41064eaf9b0"}
Oct 30 21:49:19.166512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:19.166463    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb" podStartSLOduration=2.166450171 podStartE2EDuration="2.166450171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:19.164595328 +0000 UTC m=+2872.211154464" watchObservedRunningTime="2026-10-30 21:49:19.166450171 +0000 UTC m=+2872.213009315"
Oct 30 21:49:19.194574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:19.194525    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2" podStartSLOduration=2.194511164 podStartE2EDuration="2.194511164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:19.190498509 +0000 UTC m=+2872.237057680" watchObservedRunningTime="2026-10-30 21:49:19.194511164 +0000 UTC m=+2872.241070338"
Oct 30 21:49:20.160339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.160307    4918 generic.go:356] "Generic (PLEG): container finished" podID="69611331-493d-459d-83b1-e977506590af" containerID="28c81b41f0be9d7a76f06fea787f7ec8014748a84f63cb1db4e50e2936c0cb76" exitCode=0
Oct 30 21:49:20.160820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.160408    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb" event={"ID":"69611331-493d-459d-83b1-e977506590af","Type":"ContainerDied","Data":"28c81b41f0be9d7a76f06fea787f7ec8014748a84f63cb1db4e50e2936c0cb76"}
Oct 30 21:49:20.162350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.162330    4918 generic.go:356] "Generic (PLEG): container finished" podID="0138ea77-3423-43be-abbb-404ea9e94c8c" containerID="0032c21601142c952da06f52618680406b9f93f123defcac8bedc41064eaf9b0" exitCode=0
Oct 30 21:49:20.162479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.162420    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2" event={"ID":"0138ea77-3423-43be-abbb-404ea9e94c8c","Type":"ContainerDied","Data":"0032c21601142c952da06f52618680406b9f93f123defcac8bedc41064eaf9b0"}
Oct 30 21:49:20.349732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.349695    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main" containerID="cri-o://d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5" gracePeriod=45
Oct 30 21:49:20.354625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.354591    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main" containerID="cri-o://6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1" gracePeriod=45
Oct 30 21:49:20.712498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.712468    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:49:20.718146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.718113    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:49:20.867067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.867030    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-model-cache\") pod \"e7a0864e-46fb-42d8-892e-0579659d468d\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") "
Oct 30 21:49:20.867263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.867140    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a0864e-46fb-42d8-892e-0579659d468d-kube-api-access-wnhds\" (UniqueName: \"kubernetes.io/projected/e7a0864e-46fb-42d8-892e-0579659d468d-kube-api-access-wnhds\") pod \"e7a0864e-46fb-42d8-892e-0579659d468d\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") "
Oct 30 21:49:20.867263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.867222    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tmp-dir\") pod \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") "
Oct 30 21:49:20.867263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.867258    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tls-certs\") pod \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") "
Oct 30 21:49:20.867454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.867290    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-model-cache" pod "e7a0864e-46fb-42d8-892e-0579659d468d" (UID: "e7a0864e-46fb-42d8-892e-0579659d468d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:20.867454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.867348    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-tmp-dir\") pod \"e7a0864e-46fb-42d8-892e-0579659d468d\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") "
Oct 30 21:49:20.867790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.867764    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tmp-dir" pod "d7fe75d1-2b2e-4aaa-b2b5-752796087620" (UID: "d7fe75d1-2b2e-4aaa-b2b5-752796087620"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:20.868232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.868207    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-tmp-dir" pod "e7a0864e-46fb-42d8-892e-0579659d468d" (UID: "e7a0864e-46fb-42d8-892e-0579659d468d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:20.868630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.868609    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-dshm\") pod \"e7a0864e-46fb-42d8-892e-0579659d468d\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") "
Oct 30 21:49:20.871598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.871560    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a0864e-46fb-42d8-892e-0579659d468d-kube-api-access-wnhds" pod "e7a0864e-46fb-42d8-892e-0579659d468d" (UID: "e7a0864e-46fb-42d8-892e-0579659d468d"). InnerVolumeSpecName "kube-api-access-wnhds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:20.871598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.871506    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-dshm\") pod \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") "
Oct 30 21:49:20.872730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.872708    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7a0864e-46fb-42d8-892e-0579659d468d-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a0864e-46fb-42d8-892e-0579659d468d-tls-certs\") pod \"e7a0864e-46fb-42d8-892e-0579659d468d\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") "
Oct 30 21:49:20.873058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.872922    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-home\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-home\") pod \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") "
Oct 30 21:49:20.873058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.872951    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-dshm" pod "d7fe75d1-2b2e-4aaa-b2b5-752796087620" (UID: "d7fe75d1-2b2e-4aaa-b2b5-752796087620"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:20.873058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.872971    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-model-cache\") pod \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") "
Oct 30 21:49:20.873539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.873197    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-home" pod "d7fe75d1-2b2e-4aaa-b2b5-752796087620" (UID: "d7fe75d1-2b2e-4aaa-b2b5-752796087620"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:20.873539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.873288    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7fe75d1-2b2e-4aaa-b2b5-752796087620-kube-api-access-dhz2s\" (UniqueName: \"kubernetes.io/projected/d7fe75d1-2b2e-4aaa-b2b5-752796087620-kube-api-access-dhz2s\") pod \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\" (UID: \"d7fe75d1-2b2e-4aaa-b2b5-752796087620\") "
Oct 30 21:49:20.873539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.873323    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-model-cache" pod "d7fe75d1-2b2e-4aaa-b2b5-752796087620" (UID: "d7fe75d1-2b2e-4aaa-b2b5-752796087620"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:20.873539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.873339    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-home\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-home\") pod \"e7a0864e-46fb-42d8-892e-0579659d468d\" (UID: \"e7a0864e-46fb-42d8-892e-0579659d468d\") "
Oct 30 21:49:20.873934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.873911    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-home" pod "e7a0864e-46fb-42d8-892e-0579659d468d" (UID: "e7a0864e-46fb-42d8-892e-0579659d468d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:20.875957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.875886    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7a0864e-46fb-42d8-892e-0579659d468d-tls-certs" pod "e7a0864e-46fb-42d8-892e-0579659d468d" (UID: "e7a0864e-46fb-42d8-892e-0579659d468d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:20.875957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.875924    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7fe75d1-2b2e-4aaa-b2b5-752796087620-kube-api-access-dhz2s" pod "d7fe75d1-2b2e-4aaa-b2b5-752796087620" (UID: "d7fe75d1-2b2e-4aaa-b2b5-752796087620"). InnerVolumeSpecName "kube-api-access-dhz2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:20.876170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.876152    4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:20.876262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.876176    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnhds\" (UniqueName: \"kubernetes.io/projected/e7a0864e-46fb-42d8-892e-0579659d468d-kube-api-access-wnhds\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:20.876262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.876193    4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:20.876262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.876206    4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:20.876262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.876217    4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:20.876262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.876227    4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a0864e-46fb-42d8-892e-0579659d468d-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:20.876262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.876237    4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:20.876262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.876248    4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7fe75d1-2b2e-4aaa-b2b5-752796087620-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:20.876262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.876259    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhz2s\" (UniqueName: \"kubernetes.io/projected/d7fe75d1-2b2e-4aaa-b2b5-752796087620-kube-api-access-dhz2s\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:20.876839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.876270    4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:20.876839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.876291    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tls-certs" pod "d7fe75d1-2b2e-4aaa-b2b5-752796087620" (UID: "d7fe75d1-2b2e-4aaa-b2b5-752796087620"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:20.878599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.878564    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-dshm" pod "e7a0864e-46fb-42d8-892e-0579659d468d" (UID: "e7a0864e-46fb-42d8-892e-0579659d468d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:20.977800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.977766    4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7fe75d1-2b2e-4aaa-b2b5-752796087620-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:20.977800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:20.977798    4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a0864e-46fb-42d8-892e-0579659d468d-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:21.173192 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.173163    4918 generic.go:356] "Generic (PLEG): container finished" podID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerID="6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1" exitCode=0
Oct 30 21:49:21.173686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.173226    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"
Oct 30 21:49:21.173686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.173248    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" event={"ID":"d7fe75d1-2b2e-4aaa-b2b5-752796087620","Type":"ContainerDied","Data":"6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1"}
Oct 30 21:49:21.173686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.173293    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9" event={"ID":"d7fe75d1-2b2e-4aaa-b2b5-752796087620","Type":"ContainerDied","Data":"26ed78cf7c24c50042abe7a863f14d01fae9d5118ad6ffc8fd54ed4dba5d9a3d"}
Oct 30 21:49:21.173686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.173313    4918 scope.go:122] "RemoveContainer" containerID="6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1"
Oct 30 21:49:21.185988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.185954    4918 generic.go:356] "Generic (PLEG): container finished" podID="e7a0864e-46fb-42d8-892e-0579659d468d" containerID="d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5" exitCode=0
Oct 30 21:49:21.186341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.186317    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" event={"ID":"e7a0864e-46fb-42d8-892e-0579659d468d","Type":"ContainerDied","Data":"d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5"}
Oct 30 21:49:21.186455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.186369    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"
Oct 30 21:49:21.186598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.186384    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv" event={"ID":"e7a0864e-46fb-42d8-892e-0579659d468d","Type":"ContainerDied","Data":"3b4afb33c04694814f4236e0e0f3fac53a502ede827b05af5ed335ac15c4ce58"}
Oct 30 21:49:21.205907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.205880    4918 scope.go:122] "RemoveContainer" containerID="6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1"
Oct 30 21:49:21.206335 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:49:21.206297    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1\": container with ID starting with 6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1 not found: ID does not exist" containerID="6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1"
Oct 30 21:49:21.206484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.206359    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1"} err="failed to get container status \"6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1\": rpc error: code = NotFound desc = could not find container \"6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1\": container with ID starting with 6f4bb0e5a7d889a2f125e15227752e870c8f569c11adec1507eb99bdb53919b1 not found: ID does not exist"
Oct 30 21:49:21.206484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.206456    4918 scope.go:122] "RemoveContainer" containerID="d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5"
Oct 30 21:49:21.211024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.210975    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"]
Oct 30 21:49:21.223133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.223077    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-384164/simulator-w0-384164-kserve-856dc7ddf-wndh9"]
Oct 30 21:49:21.228940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.227843    4918 scope.go:122] "RemoveContainer" containerID="d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5"
Oct 30 21:49:21.228940 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:49:21.228198    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5\": container with ID starting with d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5 not found: ID does not exist" containerID="d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5"
Oct 30 21:49:21.228940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.228233    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5"} err="failed to get container status \"d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5\": rpc error: code = NotFound desc = could not find container \"d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5\": container with ID starting with d9bfd9232040ea978e49e11e597572359ff8dd233613717661aafdcea0ace9e5 not found: ID does not exist"
Oct 30 21:49:21.230378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.230349    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"]
Oct 30 21:49:21.238153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.238127    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-384164/distinct-w0-384164-kserve-788d878db7-k9knv"]
Oct 30 21:49:21.547950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.547924    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:21.598657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.598633    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:21.715026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.714993    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69611331-493d-459d-83b1-e977506590af-kube-api-access-rfsjb\" (UniqueName: \"kubernetes.io/projected/69611331-493d-459d-83b1-e977506590af-kube-api-access-rfsjb\") pod \"69611331-493d-459d-83b1-e977506590af\" (UID: \"69611331-493d-459d-83b1-e977506590af\") "
Oct 30 21:49:21.715233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.715124    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-xks-ca\") pod \"0138ea77-3423-43be-abbb-404ea9e94c8c\" (UID: \"0138ea77-3423-43be-abbb-404ea9e94c8c\") "
Oct 30 21:49:21.715233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.715185    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69611331-493d-459d-83b1-e977506590af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69611331-493d-459d-83b1-e977506590af-maas-api-xks-ca\") pod \"69611331-493d-459d-83b1-e977506590af\" (UID: \"69611331-493d-459d-83b1-e977506590af\") "
Oct 30 21:49:21.715369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.715241    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69611331-493d-459d-83b1-e977506590af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69611331-493d-459d-83b1-e977506590af-maas-api-openshift-ca\") pod \"69611331-493d-459d-83b1-e977506590af\" (UID: \"69611331-493d-459d-83b1-e977506590af\") "
Oct 30 21:49:21.715369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.715350    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0138ea77-3423-43be-abbb-404ea9e94c8c-kube-api-access-7pfkz\" (UniqueName: \"kubernetes.io/projected/0138ea77-3423-43be-abbb-404ea9e94c8c-kube-api-access-7pfkz\") pod \"0138ea77-3423-43be-abbb-404ea9e94c8c\" (UID: \"0138ea77-3423-43be-abbb-404ea9e94c8c\") "
Oct 30 21:49:21.715564 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.715536    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-openshift-ca\") pod \"0138ea77-3423-43be-abbb-404ea9e94c8c\" (UID: \"0138ea77-3423-43be-abbb-404ea9e94c8c\") "
Oct 30 21:49:21.715764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.715704    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69611331-493d-459d-83b1-e977506590af-maas-api-openshift-ca" pod "69611331-493d-459d-83b1-e977506590af" (UID: "69611331-493d-459d-83b1-e977506590af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:21.716643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.716613    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69611331-493d-459d-83b1-e977506590af-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:21.716775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.716646    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-openshift-ca" pod "0138ea77-3423-43be-abbb-404ea9e94c8c" (UID: "0138ea77-3423-43be-abbb-404ea9e94c8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:21.717986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.717955    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69611331-493d-459d-83b1-e977506590af-maas-api-xks-ca" pod "69611331-493d-459d-83b1-e977506590af" (UID: "69611331-493d-459d-83b1-e977506590af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:21.718148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.718125    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69611331-493d-459d-83b1-e977506590af-kube-api-access-rfsjb" pod "69611331-493d-459d-83b1-e977506590af" (UID: "69611331-493d-459d-83b1-e977506590af"). InnerVolumeSpecName "kube-api-access-rfsjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:21.718735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.718716    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0138ea77-3423-43be-abbb-404ea9e94c8c-kube-api-access-7pfkz" pod "0138ea77-3423-43be-abbb-404ea9e94c8c" (UID: "0138ea77-3423-43be-abbb-404ea9e94c8c"). InnerVolumeSpecName "kube-api-access-7pfkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:21.719129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.719114    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-xks-ca" pod "0138ea77-3423-43be-abbb-404ea9e94c8c" (UID: "0138ea77-3423-43be-abbb-404ea9e94c8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:21.817915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.817837    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:21.817915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.817864    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69611331-493d-459d-83b1-e977506590af-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:21.817915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.817874    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pfkz\" (UniqueName: \"kubernetes.io/projected/0138ea77-3423-43be-abbb-404ea9e94c8c-kube-api-access-7pfkz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:21.817915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.817885    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0138ea77-3423-43be-abbb-404ea9e94c8c-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:21.817915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:21.817893    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfsjb\" (UniqueName: \"kubernetes.io/projected/69611331-493d-459d-83b1-e977506590af-kube-api-access-rfsjb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:22.201220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:22.201145    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"
Oct 30 21:49:22.201220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:22.201190    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb" event={"ID":"69611331-493d-459d-83b1-e977506590af","Type":"ContainerDied","Data":"5850bc6333bd174f94f16bfbcc8bbd857384a13783821cc3b7103fe9a9b74f7b"}
Oct 30 21:49:22.201699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:22.201243    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5850bc6333bd174f94f16bfbcc8bbd857384a13783821cc3b7103fe9a9b74f7b"
Oct 30 21:49:22.202870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:22.202845    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2" event={"ID":"0138ea77-3423-43be-abbb-404ea9e94c8c","Type":"ContainerDied","Data":"c0a3d06417ca5152576d38158e19af21e0349357fbc392b36ad461e825275da4"}
Oct 30 21:49:22.202870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:22.202862    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"
Oct 30 21:49:22.203060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:22.202885    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0a3d06417ca5152576d38158e19af21e0349357fbc392b36ad461e825275da4"
Oct 30 21:49:23.083230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.083198    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" path="/var/lib/kubelet/pods/d7fe75d1-2b2e-4aaa-b2b5-752796087620/volumes"
Oct 30 21:49:23.084007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.083987    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" path="/var/lib/kubelet/pods/e7a0864e-46fb-42d8-892e-0579659d468d/volumes"
Oct 30 21:49:23.254578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.254547    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"]
Oct 30 21:49:23.255819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.255801    4918 cpu_manager.go:414] "containerMap: removing container" podUID="0138ea77-3423-43be-abbb-404ea9e94c8c" containerName="revoke-keys"
Oct 30 21:49:23.255874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.255822    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0138ea77-3423-43be-abbb-404ea9e94c8c" containerName="revoke-keys"
Oct 30 21:49:23.255924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.255895    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main"
Oct 30 21:49:23.255924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.255904    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main"
Oct 30 21:49:23.255998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.255921    4918 cpu_manager.go:414] "containerMap: removing container" podUID="69611331-493d-459d-83b1-e977506590af" containerName="revoke-keys"
Oct 30 21:49:23.255998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.255929    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69611331-493d-459d-83b1-e977506590af" containerName="revoke-keys"
Oct 30 21:49:23.255998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.255941    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main"
Oct 30 21:49:23.255998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.255950    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main"
Oct 30 21:49:23.256147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.256136    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7fe75d1-2b2e-4aaa-b2b5-752796087620" containerName="main"
Oct 30 21:49:23.256187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.256151    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="69611331-493d-459d-83b1-e977506590af" containerName="revoke-keys"
Oct 30 21:49:23.256187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.256160    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a0864e-46fb-42d8-892e-0579659d468d" containerName="main"
Oct 30 21:49:23.256187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.256175    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0138ea77-3423-43be-abbb-404ea9e94c8c" containerName="revoke-keys"
Oct 30 21:49:23.261802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.261783    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"
Oct 30 21:49:23.291944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.291912    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"]
Oct 30 21:49:23.307469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.295123    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-adf308\""
Oct 30 21:49:23.321381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.321354    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"]
Oct 30 21:49:23.326385 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.326368    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"
Oct 30 21:49:23.359760 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.359724    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755a5f2c-b99e-4c7b-893a-d346b9f92d81-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f\" (UID: \"755a5f2c-b99e-4c7b-893a-d346b9f92d81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"
Oct 30 21:49:23.359981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.359916    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jllrl\" (UniqueName: \"kubernetes.io/projected/755a5f2c-b99e-4c7b-893a-d346b9f92d81-kube-api-access-jllrl\") pod \"payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f\" (UID: \"755a5f2c-b99e-4c7b-893a-d346b9f92d81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"
Oct 30 21:49:23.365013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.364988    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"]
Oct 30 21:49:23.401131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.388622    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-adf308-dockercfg-9mt8r\""
Oct 30 21:49:23.517415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.475996    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755a5f2c-b99e-4c7b-893a-d346b9f92d81-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f\" (UID: \"755a5f2c-b99e-4c7b-893a-d346b9f92d81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"
Oct 30 21:49:23.517415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.476162    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgr94\" (UniqueName: \"kubernetes.io/projected/4db3211f-c998-428e-9364-88befce1cf71-kube-api-access-vgr94\") pod \"payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc\" (UID: \"4db3211f-c998-428e-9364-88befce1cf71\") " pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"
Oct 30 21:49:23.517415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.476216    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db3211f-c998-428e-9364-88befce1cf71-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc\" (UID: \"4db3211f-c998-428e-9364-88befce1cf71\") " pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"
Oct 30 21:49:23.517415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.476258    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jllrl\" (UniqueName: \"kubernetes.io/projected/755a5f2c-b99e-4c7b-893a-d346b9f92d81-kube-api-access-jllrl\") pod \"payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f\" (UID: \"755a5f2c-b99e-4c7b-893a-d346b9f92d81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"
Oct 30 21:49:23.517415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.477858    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755a5f2c-b99e-4c7b-893a-d346b9f92d81-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f\" (UID: \"755a5f2c-b99e-4c7b-893a-d346b9f92d81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"
Oct 30 21:49:23.618465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.617164    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jllrl\" (UniqueName: \"kubernetes.io/projected/755a5f2c-b99e-4c7b-893a-d346b9f92d81-kube-api-access-jllrl\") pod \"payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f\" (UID: \"755a5f2c-b99e-4c7b-893a-d346b9f92d81\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"
Oct 30 21:49:23.630416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.618692    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db3211f-c998-428e-9364-88befce1cf71-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc\" (UID: \"4db3211f-c998-428e-9364-88befce1cf71\") " pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"
Oct 30 21:49:23.630416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.619258    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgr94\" (UniqueName: \"kubernetes.io/projected/4db3211f-c998-428e-9364-88befce1cf71-kube-api-access-vgr94\") pod \"payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc\" (UID: \"4db3211f-c998-428e-9364-88befce1cf71\") " pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"
Oct 30 21:49:23.647417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.645880    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db3211f-c998-428e-9364-88befce1cf71-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc\" (UID: \"4db3211f-c998-428e-9364-88befce1cf71\") " pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"
Oct 30 21:49:23.706415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.706290    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgr94\" (UniqueName: \"kubernetes.io/projected/4db3211f-c998-428e-9364-88befce1cf71-kube-api-access-vgr94\") pod \"payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc\" (UID: \"4db3211f-c998-428e-9364-88befce1cf71\") " pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"
Oct 30 21:49:23.877065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.876863    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"
Oct 30 21:49:23.942659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:23.942623    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"
Oct 30 21:49:24.058453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:24.058417    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g" podUID="036bea86-70eb-48e1-b108-b71928175a68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:49:24.190358 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:49:24.190320    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755a5f2c_b99e_4c7b_893a_d346b9f92d81.slice/crio-e52613bf720faa914fe32129b8b9bae1e7af494b702214f16d2e0f8b119350a2 WatchSource:0}: Error finding container e52613bf720faa914fe32129b8b9bae1e7af494b702214f16d2e0f8b119350a2: Status 404 returned error can't find the container with id e52613bf720faa914fe32129b8b9bae1e7af494b702214f16d2e0f8b119350a2
Oct 30 21:49:24.191608 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:24.191575    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"]
Oct 30 21:49:24.225925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:24.225860    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f" event={"ID":"755a5f2c-b99e-4c7b-893a-d346b9f92d81","Type":"ContainerStarted","Data":"e52613bf720faa914fe32129b8b9bae1e7af494b702214f16d2e0f8b119350a2"}
Oct 30 21:49:24.289601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:24.288757    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"]
Oct 30 21:49:24.303710 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:49:24.303629    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db3211f_c998_428e_9364_88befce1cf71.slice/crio-24668ed08aa960c2b8404f1c4d868f14e3c039710956e831370d2017dd111aa1 WatchSource:0}: Error finding container 24668ed08aa960c2b8404f1c4d868f14e3c039710956e831370d2017dd111aa1: Status 404 returned error can't find the container with id 24668ed08aa960c2b8404f1c4d868f14e3c039710956e831370d2017dd111aa1
Oct 30 21:49:25.238329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:25.238290    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f" event={"ID":"755a5f2c-b99e-4c7b-893a-d346b9f92d81","Type":"ContainerStarted","Data":"7fce3c830e08331123298305c4bec06d9de0b27b04eee3d3f16a7385d31b0c47"}
Oct 30 21:49:25.238537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:25.238358    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"
Oct 30 21:49:25.240193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:25.240162    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc" event={"ID":"4db3211f-c998-428e-9364-88befce1cf71","Type":"ContainerStarted","Data":"bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b"}
Oct 30 21:49:25.240345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:25.240207    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc" event={"ID":"4db3211f-c998-428e-9364-88befce1cf71","Type":"ContainerStarted","Data":"24668ed08aa960c2b8404f1c4d868f14e3c039710956e831370d2017dd111aa1"}
Oct 30 21:49:25.240345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:25.240229    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"
Oct 30 21:49:25.282215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:25.282167    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f" podStartSLOduration=2.282152978 podStartE2EDuration="2.282152978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:25.278868477 +0000 UTC m=+2878.325427639" watchObservedRunningTime="2026-10-30 21:49:25.282152978 +0000 UTC m=+2878.328712174"
Oct 30 21:49:25.352217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:25.352165    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc" podStartSLOduration=2.352151026 podStartE2EDuration="2.352151026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:25.347925322 +0000 UTC m=+2878.394484488" watchObservedRunningTime="2026-10-30 21:49:25.352151026 +0000 UTC m=+2878.398710170"
Oct 30 21:49:29.541735 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:49:29.541694    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:49:30.053413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:30.053112    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"]
Oct 30 21:49:30.062759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:30.062729    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da6cadf-dhgf5"]
Oct 30 21:49:31.111266 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:31.111055    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745e7408-a27f-482f-ba79-8d8741c0ef7d" path="/var/lib/kubelet/pods/745e7408-a27f-482f-ba79-8d8741c0ef7d/volumes"
Oct 30 21:49:34.057509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:34.057466    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g" podUID="036bea86-70eb-48e1-b108-b71928175a68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:49:36.107007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.106976    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"
Oct 30 21:49:36.199812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.199779    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036bea86-70eb-48e1-b108-b71928175a68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036bea86-70eb-48e1-b108-b71928175a68-plugins-config-volume\") pod \"036bea86-70eb-48e1-b108-b71928175a68\" (UID: \"036bea86-70eb-48e1-b108-b71928175a68\") "
Oct 30 21:49:36.200183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.200162    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036bea86-70eb-48e1-b108-b71928175a68-kube-api-access-j92xg\" (UniqueName: \"kubernetes.io/projected/036bea86-70eb-48e1-b108-b71928175a68-kube-api-access-j92xg\") pod \"036bea86-70eb-48e1-b108-b71928175a68\" (UID: \"036bea86-70eb-48e1-b108-b71928175a68\") "
Oct 30 21:49:36.200273 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.200197    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036bea86-70eb-48e1-b108-b71928175a68-plugins-config-volume" pod "036bea86-70eb-48e1-b108-b71928175a68" (UID: "036bea86-70eb-48e1-b108-b71928175a68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:36.200962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.200936    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036bea86-70eb-48e1-b108-b71928175a68-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:36.203255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.203161    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036bea86-70eb-48e1-b108-b71928175a68-kube-api-access-j92xg" pod "036bea86-70eb-48e1-b108-b71928175a68" (UID: "036bea86-70eb-48e1-b108-b71928175a68"). InnerVolumeSpecName "kube-api-access-j92xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:36.252708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.252681    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"
Oct 30 21:49:36.252854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.252756    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"
Oct 30 21:49:36.306492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.303247    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j92xg\" (UniqueName: \"kubernetes.io/projected/036bea86-70eb-48e1-b108-b71928175a68-kube-api-access-j92xg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:49:36.544449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.544350    4918 generic.go:356] "Generic (PLEG): container finished" podID="036bea86-70eb-48e1-b108-b71928175a68" containerID="799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb" exitCode=0
Oct 30 21:49:36.544774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.544743    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g" event={"ID":"036bea86-70eb-48e1-b108-b71928175a68","Type":"ContainerDied","Data":"799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb"}
Oct 30 21:49:36.544892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.544815    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g" event={"ID":"036bea86-70eb-48e1-b108-b71928175a68","Type":"ContainerDied","Data":"a84cb72fbd5f6f8c0ee0fa2ecadf0e5c5fa3ae71d6a22f6da8502582346759d4"}
Oct 30 21:49:36.544892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.544847    4918 scope.go:122] "RemoveContainer" containerID="799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb"
Oct 30 21:49:36.545109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.544755    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"
Oct 30 21:49:36.562938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.562917    4918 scope.go:122] "RemoveContainer" containerID="799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb"
Oct 30 21:49:36.563292 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:49:36.563226    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb\": container with ID starting with 799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb not found: ID does not exist" containerID="799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb"
Oct 30 21:49:36.563292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.563269    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb"} err="failed to get container status \"799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb\": rpc error: code = NotFound desc = could not find container \"799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb\": container with ID starting with 799b59e63e1e7b49ad8f755dc4713a8c5d0f99ff2550ce334ee7249d15bd87fb not found: ID does not exist"
Oct 30 21:49:36.580571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.580534    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"]
Oct 30 21:49:36.585533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:36.585510    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-384164-b86fb9979-j4j8g"]
Oct 30 21:49:37.083709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:37.083677    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036bea86-70eb-48e1-b108-b71928175a68" path="/var/lib/kubelet/pods/036bea86-70eb-48e1-b108-b71928175a68/volumes"
Oct 30 21:49:40.148075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:40.146564    4918 patch_prober.go:29] interesting pod/e2e-inf-b-001667-openshift-default-c686545cc-97cd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body=
Oct 30 21:49:40.148075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:40.146665    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused"
Oct 30 21:49:40.587065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:40.587033    4918 generic.go:356] "Generic (PLEG): container finished" podID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerID="8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf" exitCode=0
Oct 30 21:49:40.587249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:40.587103    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" event={"ID":"119bae8a-d416-4701-bfc6-eeabc9db6ef9","Type":"ContainerDied","Data":"8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf"}
Oct 30 21:49:40.587682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:40.587662    4918 scope.go:122] "RemoveContainer" containerID="8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf"
Oct 30 21:49:40.589728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:40.589695    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:49:40.589885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:40.589783    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:49:40.589885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:40.589845    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:49:41.030937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:41.030899    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"]
Oct 30 21:49:41.035726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:41.035693    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa74ed49-7x9xh"]
Oct 30 21:49:41.081733 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:41.081702    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb" path="/var/lib/kubelet/pods/9f2eeeff-ccbe-41df-aaf5-5cabfc8965bb/volumes"
Oct 30 21:49:41.598823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:41.598787    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" event={"ID":"119bae8a-d416-4701-bfc6-eeabc9db6ef9","Type":"ContainerStarted","Data":"d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b"}
Oct 30 21:49:41.915469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:41.913317    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:49:41.915469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:41.913523    4918 patch_prober.go:29] interesting pod/e2e-inf-b-001667-openshift-default-c686545cc-97cd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body=
Oct 30 21:49:41.915469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:41.913796    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused"
Oct 30 21:49:42.922415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:42.920208    4918 patch_prober.go:29] interesting pod/e2e-inf-b-001667-openshift-default-c686545cc-97cd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body=
Oct 30 21:49:42.922415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:42.920268    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused"
Oct 30 21:49:43.910416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:43.910248    4918 patch_prober.go:29] interesting pod/e2e-inf-b-001667-openshift-default-c686545cc-97cd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body=
Oct 30 21:49:43.910416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:43.910307    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused"
Oct 30 21:49:44.911419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:44.910755    4918 patch_prober.go:29] interesting pod/e2e-inf-b-001667-openshift-default-c686545cc-97cd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body=
Oct 30 21:49:44.911419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:44.910812    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused"
Oct 30 21:49:45.912302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:45.911753    4918 patch_prober.go:29] interesting pod/e2e-inf-b-001667-openshift-default-c686545cc-97cd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body=
Oct 30 21:49:45.912302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:45.911812    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused"
Oct 30 21:49:46.918464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:46.914755    4918 patch_prober.go:29] interesting pod/e2e-inf-b-001667-openshift-default-c686545cc-97cd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body=
Oct 30 21:49:46.918464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:46.914817    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused"
Oct 30 21:49:47.990515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:47.990486    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:49:47.991031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:47.991002    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:49:47.991837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:47.991820    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:49:55.468414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:55.465668    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"]
Oct 30 21:49:55.468414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:49:55.465933    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" podUID="f82566da-633b-47a2-8b06-389302d80839" containerName="istio-proxy" containerID="cri-o://5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073" gracePeriod=30
Oct 30 21:49:59.162586 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:49:59.162552    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:50:00.787849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.787824    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:50:00.849723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.849692    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-data\") pod \"f82566da-633b-47a2-8b06-389302d80839\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") "
Oct 30 21:50:00.849903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.849870    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f82566da-633b-47a2-8b06-389302d80839-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82566da-633b-47a2-8b06-389302d80839-istio-podinfo\") pod \"f82566da-633b-47a2-8b06-389302d80839\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") "
Oct 30 21:50:00.849961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.849916    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-credential-socket\") pod \"f82566da-633b-47a2-8b06-389302d80839\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") "
Oct 30 21:50:00.850002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.849969    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-kube-api-access-t2jfw\" (UniqueName: \"kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-kube-api-access-t2jfw\") pod \"f82566da-633b-47a2-8b06-389302d80839\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") "
Oct 30 21:50:00.850065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.850021    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-envoy\") pod \"f82566da-633b-47a2-8b06-389302d80839\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") "
Oct 30 21:50:00.850065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.850025    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-data" pod "f82566da-633b-47a2-8b06-389302d80839" (UID: "f82566da-633b-47a2-8b06-389302d80839"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:00.850170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.850154    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f82566da-633b-47a2-8b06-389302d80839-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82566da-633b-47a2-8b06-389302d80839-istiod-ca-cert\") pod \"f82566da-633b-47a2-8b06-389302d80839\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") "
Oct 30 21:50:00.850236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.850225    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-socket\") pod \"f82566da-633b-47a2-8b06-389302d80839\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") "
Oct 30 21:50:00.850293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.850277    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-istio-token\" (UniqueName: \"kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-istio-token\") pod \"f82566da-633b-47a2-8b06-389302d80839\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") "
Oct 30 21:50:00.850353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.850305    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-credential-socket" pod "f82566da-633b-47a2-8b06-389302d80839" (UID: "f82566da-633b-47a2-8b06-389302d80839"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:00.850353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.850325    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-certs\") pod \"f82566da-633b-47a2-8b06-389302d80839\" (UID: \"f82566da-633b-47a2-8b06-389302d80839\") "
Oct 30 21:50:00.850544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.850509    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-socket" pod "f82566da-633b-47a2-8b06-389302d80839" (UID: "f82566da-633b-47a2-8b06-389302d80839"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:00.850623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.850601    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82566da-633b-47a2-8b06-389302d80839-istiod-ca-cert" pod "f82566da-633b-47a2-8b06-389302d80839" (UID: "f82566da-633b-47a2-8b06-389302d80839"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:00.850785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.850763    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-certs" pod "f82566da-633b-47a2-8b06-389302d80839" (UID: "f82566da-633b-47a2-8b06-389302d80839"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:00.851539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.851426    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f82566da-633b-47a2-8b06-389302d80839-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.851539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.851453    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.851539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.851469    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.851539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.851484    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.851539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.851498    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.853252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.852864    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-kube-api-access-t2jfw" pod "f82566da-633b-47a2-8b06-389302d80839" (UID: "f82566da-633b-47a2-8b06-389302d80839"). InnerVolumeSpecName "kube-api-access-t2jfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:00.853252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.852979    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f82566da-633b-47a2-8b06-389302d80839-istio-podinfo" pod "f82566da-633b-47a2-8b06-389302d80839" (UID: "f82566da-633b-47a2-8b06-389302d80839"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:50:00.853252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.853057    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-istio-token" pod "f82566da-633b-47a2-8b06-389302d80839" (UID: "f82566da-633b-47a2-8b06-389302d80839"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:00.853589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.853565    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-envoy" pod "f82566da-633b-47a2-8b06-389302d80839" (UID: "f82566da-633b-47a2-8b06-389302d80839"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:00.953641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.953607    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f82566da-633b-47a2-8b06-389302d80839-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.953641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.953639    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2jfw\" (UniqueName: \"kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-kube-api-access-t2jfw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.953641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.953650    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f82566da-633b-47a2-8b06-389302d80839-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.953945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.953663    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f82566da-633b-47a2-8b06-389302d80839-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:00.984804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.984767    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"]
Oct 30 21:50:00.985972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.985953    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f82566da-633b-47a2-8b06-389302d80839" containerName="istio-proxy"
Oct 30 21:50:00.986072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.985976    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82566da-633b-47a2-8b06-389302d80839" containerName="istio-proxy"
Oct 30 21:50:00.986072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.985995    4918 cpu_manager.go:414] "containerMap: removing container" podUID="036bea86-70eb-48e1-b108-b71928175a68" containerName="payload-processing"
Oct 30 21:50:00.986072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.986000    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036bea86-70eb-48e1-b108-b71928175a68" containerName="payload-processing"
Oct 30 21:50:00.986221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.986198    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82566da-633b-47a2-8b06-389302d80839" containerName="istio-proxy"
Oct 30 21:50:00.986221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.986219    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="036bea86-70eb-48e1-b108-b71928175a68" containerName="payload-processing"
Oct 30 21:50:00.987224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.987196    4918 generic.go:356] "Generic (PLEG): container finished" podID="f82566da-633b-47a2-8b06-389302d80839" containerID="5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073" exitCode=0
Oct 30 21:50:00.987351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.987315    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"
Oct 30 21:50:00.990746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.990725    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" event={"ID":"f82566da-633b-47a2-8b06-389302d80839","Type":"ContainerDied","Data":"5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073"}
Oct 30 21:50:00.990838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.990758    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c" event={"ID":"f82566da-633b-47a2-8b06-389302d80839","Type":"ContainerDied","Data":"66986ada6549f4414d9e111462a1a8f87ac42d09aced17d5856602a50e761ff6"}
Oct 30 21:50:00.990838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.990783    4918 scope.go:122] "RemoveContainer" containerID="5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073"
Oct 30 21:50:00.990913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.990860    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:00.997242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:00.997220    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8431de-openshift-default-dockercfg-w6wgl\""
Oct 30 21:50:01.004072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.004048    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"]
Oct 30 21:50:01.012476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.012455    4918 scope.go:122] "RemoveContainer" containerID="5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073"
Oct 30 21:50:01.013081 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:50:01.013055    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073\": container with ID starting with 5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073 not found: ID does not exist" containerID="5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073"
Oct 30 21:50:01.013187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.013130    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073"} err="failed to get container status \"5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073\": rpc error: code = NotFound desc = could not find container \"5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073\": container with ID starting with 5d3da7b5c0f0b559ac64d16d296dc847f29bf05ed1cc9f549b6e046ed897e073 not found: ID does not exist"
Oct 30 21:50:01.043383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.043352    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"]
Oct 30 21:50:01.051907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.051879    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-384164-openshift-default-c9776b589-vbz8c"]
Oct 30 21:50:01.092784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.092751    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6733d22f-3aa0-4a87-949f-14bed46f42ff-istiod-ca-cert\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.093224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.093180    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-credential-socket\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.106464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.093663    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-token\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.106787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.106766    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-data\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.108431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.108404    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2x9z\" (UniqueName: \"kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-kube-api-access-k2x9z\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.108730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.108713    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-socket\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.109161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.109139    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-envoy\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.109540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.109519    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-podinfo\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.109795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.109774    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-certs\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.149270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.149232    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82566da-633b-47a2-8b06-389302d80839" path="/var/lib/kubelet/pods/f82566da-633b-47a2-8b06-389302d80839/volumes"
Oct 30 21:50:01.211849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.211810    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-socket\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.212053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.211980    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-envoy\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.212208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.212180    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-podinfo\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.212288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.212226    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-socket\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.212359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.212306    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-certs\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.212491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.212383    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6733d22f-3aa0-4a87-949f-14bed46f42ff-istiod-ca-cert\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.212562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.212543    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-credential-socket\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.212623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.212590    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-token\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.212695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.212679    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-certs\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.212757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.212725    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-data\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.212815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.212785    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2x9z\" (UniqueName: \"kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-kube-api-access-k2x9z\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.212874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.212860    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-credential-socket\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.213059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.213033    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6733d22f-3aa0-4a87-949f-14bed46f42ff-istiod-ca-cert\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.213194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.213033    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-data\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.214569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.214549    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-envoy\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.214899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.214881    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-podinfo\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.222455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.222433    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2x9z\" (UniqueName: \"kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-kube-api-access-k2x9z\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.224272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.224253    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-token\") pod \"e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") " pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.325506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.325420    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:01.619091 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:50:01.619045    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6733d22f_3aa0_4a87_949f_14bed46f42ff.slice/crio-e1371ee7eb75ae4cc77f47d8acf74ce17b83bec7b721811e7c420adf9389546e WatchSource:0}: Error finding container e1371ee7eb75ae4cc77f47d8acf74ce17b83bec7b721811e7c420adf9389546e: Status 404 returned error can't find the container with id e1371ee7eb75ae4cc77f47d8acf74ce17b83bec7b721811e7c420adf9389546e
Oct 30 21:50:01.619959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.619936    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"]
Oct 30 21:50:01.621900 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.621866    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:50:01.622014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.621930    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:50:01.622014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:01.621959    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:50:02.008782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:02.008742    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" event={"ID":"6733d22f-3aa0-4a87-949f-14bed46f42ff","Type":"ContainerStarted","Data":"e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1"}
Oct 30 21:50:02.009254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:02.008796    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" event={"ID":"6733d22f-3aa0-4a87-949f-14bed46f42ff","Type":"ContainerStarted","Data":"e1371ee7eb75ae4cc77f47d8acf74ce17b83bec7b721811e7c420adf9389546e"}
Oct 30 21:50:02.031906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:02.031853    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" podStartSLOduration=2.031838898 podStartE2EDuration="2.031838898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:02.02903853 +0000 UTC m=+2915.075597702" watchObservedRunningTime="2026-10-30 21:50:02.031838898 +0000 UTC m=+2915.078398042"
Oct 30 21:50:02.326113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:02.326033    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:02.327506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:02.327480    4918 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body=
Oct 30 21:50:02.327664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:02.327537    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused"
Oct 30 21:50:03.326167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:03.326126    4918 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body=
Oct 30 21:50:03.326659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:03.326182    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused"
Oct 30 21:50:04.330752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:04.330717    4918 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body=
Oct 30 21:50:04.331275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:04.330783    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused"
Oct 30 21:50:05.325952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:05.325917    4918 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body=
Oct 30 21:50:05.326158 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:05.325985    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused"
Oct 30 21:50:06.326546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:06.326514    4918 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body=
Oct 30 21:50:06.327044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:06.327019    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused"
Oct 30 21:50:07.326205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:07.326121    4918 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body=
Oct 30 21:50:07.326205 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:07.326176    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused"
Oct 30 21:50:08.326340 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:08.326302    4918 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body=
Oct 30 21:50:08.326860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:08.326381    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused"
Oct 30 21:50:09.343355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:09.342037    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:09.343909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:09.343713    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:09.344636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:09.344615    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:50:24.023357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.023319    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"]
Oct 30 21:50:24.028020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.028003    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"
Oct 30 21:50:24.034117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.034088    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e8d315\""
Oct 30 21:50:24.034532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.034499    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e8d315-dockercfg-p8tmz\""
Oct 30 21:50:24.042382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.042355    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"]
Oct 30 21:50:24.142112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.142045    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px\" (UID: \"d5cc4d1c-e96c-4df3-8b99-99257ffaf828\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"
Oct 30 21:50:24.142457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.142431    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgq6t\" (UniqueName: \"kubernetes.io/projected/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-kube-api-access-tgq6t\") pod \"payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px\" (UID: \"d5cc4d1c-e96c-4df3-8b99-99257ffaf828\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"
Oct 30 21:50:24.279469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.278368    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px\" (UID: \"d5cc4d1c-e96c-4df3-8b99-99257ffaf828\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"
Oct 30 21:50:24.279469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.278440    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgq6t\" (UniqueName: \"kubernetes.io/projected/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-kube-api-access-tgq6t\") pod \"payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px\" (UID: \"d5cc4d1c-e96c-4df3-8b99-99257ffaf828\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"
Oct 30 21:50:24.279469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.279358    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px\" (UID: \"d5cc4d1c-e96c-4df3-8b99-99257ffaf828\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"
Oct 30 21:50:24.600480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.570196    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgq6t\" (UniqueName: \"kubernetes.io/projected/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-kube-api-access-tgq6t\") pod \"payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px\" (UID: \"d5cc4d1c-e96c-4df3-8b99-99257ffaf828\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"
Oct 30 21:50:24.646858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.646818    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"
Oct 30 21:50:24.946985 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:50:24.946957    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cc4d1c_e96c_4df3_8b99_99257ffaf828.slice/crio-cf416ee6e51d232fb801e7227180efb8ea892f7fc1a2d5ec7271a50a81a51863 WatchSource:0}: Error finding container cf416ee6e51d232fb801e7227180efb8ea892f7fc1a2d5ec7271a50a81a51863: Status 404 returned error can't find the container with id cf416ee6e51d232fb801e7227180efb8ea892f7fc1a2d5ec7271a50a81a51863
Oct 30 21:50:24.948289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:24.948268    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"]
Oct 30 21:50:25.670055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:25.670022    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px" event={"ID":"d5cc4d1c-e96c-4df3-8b99-99257ffaf828","Type":"ContainerStarted","Data":"713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1"}
Oct 30 21:50:25.670562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:25.670075    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px" event={"ID":"d5cc4d1c-e96c-4df3-8b99-99257ffaf828","Type":"ContainerStarted","Data":"cf416ee6e51d232fb801e7227180efb8ea892f7fc1a2d5ec7271a50a81a51863"}
Oct 30 21:50:25.670562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:25.670228    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"
Oct 30 21:50:25.701874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:25.701804    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px" podStartSLOduration=2.701791071 podStartE2EDuration="2.701791071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:25.700958796 +0000 UTC m=+2938.747517967" watchObservedRunningTime="2026-10-30 21:50:25.701791071 +0000 UTC m=+2938.748350216"
Oct 30 21:50:26.340488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:26.339360    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"]
Oct 30 21:50:26.340853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:26.340678    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerName="istio-proxy" containerID="cri-o://88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079" gracePeriod=30
Oct 30 21:50:29.207973 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:50:29.207934    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:50:31.647719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.647690    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:50:31.730055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.730021    4918 generic.go:356] "Generic (PLEG): container finished" podID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerID="88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079" exitCode=0
Oct 30 21:50:31.730247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.730073    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"
Oct 30 21:50:31.730247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.730102    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" event={"ID":"22dede67-e6f7-4577-a8c6-8c4c671814d1","Type":"ContainerDied","Data":"88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079"}
Oct 30 21:50:31.730247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.730175    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f" event={"ID":"22dede67-e6f7-4577-a8c6-8c4c671814d1","Type":"ContainerDied","Data":"8d50e89bc91f7fe7fbe574a4da3bc6f6fbdbfee28cf99a3c478c5ff500834c94"}
Oct 30 21:50:31.730247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.730197    4918 scope.go:122] "RemoveContainer" containerID="88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079"
Oct 30 21:50:31.742697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.742676    4918 scope.go:122] "RemoveContainer" containerID="88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079"
Oct 30 21:50:31.743029 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:50:31.743001    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079\": container with ID starting with 88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079 not found: ID does not exist" containerID="88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079"
Oct 30 21:50:31.743143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.743039    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079"} err="failed to get container status \"88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079\": rpc error: code = NotFound desc = could not find container \"88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079\": container with ID starting with 88e8d3fd62b899d2bd3905dfaf107360c62e157e0c526e981a548c3e8e628079 not found: ID does not exist"
Oct 30 21:50:31.797709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.797678    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-socket\") pod \"22dede67-e6f7-4577-a8c6-8c4c671814d1\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") "
Oct 30 21:50:31.798018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.797857    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-kube-api-access-cgjj5\" (UniqueName: \"kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-kube-api-access-cgjj5\") pod \"22dede67-e6f7-4577-a8c6-8c4c671814d1\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") "
Oct 30 21:50:31.798018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.797963    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-certs\") pod \"22dede67-e6f7-4577-a8c6-8c4c671814d1\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") "
Oct 30 21:50:31.798018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.797996    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-socket" pod "22dede67-e6f7-4577-a8c6-8c4c671814d1" (UID: "22dede67-e6f7-4577-a8c6-8c4c671814d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:31.798255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.798041    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-token\" (UniqueName: \"kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-token\") pod \"22dede67-e6f7-4577-a8c6-8c4c671814d1\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") "
Oct 30 21:50:31.798255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.798069    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-podinfo\") pod \"22dede67-e6f7-4577-a8c6-8c4c671814d1\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") "
Oct 30 21:50:31.798255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.798131    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-credential-socket\") pod \"22dede67-e6f7-4577-a8c6-8c4c671814d1\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") "
Oct 30 21:50:31.798255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.798200    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22dede67-e6f7-4577-a8c6-8c4c671814d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dede67-e6f7-4577-a8c6-8c4c671814d1-istiod-ca-cert\") pod \"22dede67-e6f7-4577-a8c6-8c4c671814d1\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") "
Oct 30 21:50:31.798657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.798370    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-envoy\") pod \"22dede67-e6f7-4577-a8c6-8c4c671814d1\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") "
Oct 30 21:50:31.798657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.798434    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-data\") pod \"22dede67-e6f7-4577-a8c6-8c4c671814d1\" (UID: \"22dede67-e6f7-4577-a8c6-8c4c671814d1\") "
Oct 30 21:50:31.798657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.798456    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-certs" pod "22dede67-e6f7-4577-a8c6-8c4c671814d1" (UID: "22dede67-e6f7-4577-a8c6-8c4c671814d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:31.798794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.798718    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-credential-socket" pod "22dede67-e6f7-4577-a8c6-8c4c671814d1" (UID: "22dede67-e6f7-4577-a8c6-8c4c671814d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:31.799298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.799155    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dede67-e6f7-4577-a8c6-8c4c671814d1-istiod-ca-cert" pod "22dede67-e6f7-4577-a8c6-8c4c671814d1" (UID: "22dede67-e6f7-4577-a8c6-8c4c671814d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:31.799298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.799144    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-data" pod "22dede67-e6f7-4577-a8c6-8c4c671814d1" (UID: "22dede67-e6f7-4577-a8c6-8c4c671814d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:31.799585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.799558    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:31.799709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.799589    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:31.799709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.799604    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22dede67-e6f7-4577-a8c6-8c4c671814d1-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:31.799709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.799618    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:31.799709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.799631    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:31.801366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.801336    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-token" pod "22dede67-e6f7-4577-a8c6-8c4c671814d1" (UID: "22dede67-e6f7-4577-a8c6-8c4c671814d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:31.801659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.801638    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-kube-api-access-cgjj5" pod "22dede67-e6f7-4577-a8c6-8c4c671814d1" (UID: "22dede67-e6f7-4577-a8c6-8c4c671814d1"). InnerVolumeSpecName "kube-api-access-cgjj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:31.801968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.801945    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-envoy" pod "22dede67-e6f7-4577-a8c6-8c4c671814d1" (UID: "22dede67-e6f7-4577-a8c6-8c4c671814d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:31.802088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.801986    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-podinfo" pod "22dede67-e6f7-4577-a8c6-8c4c671814d1" (UID: "22dede67-e6f7-4577-a8c6-8c4c671814d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:50:31.901821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.901731    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgjj5\" (UniqueName: \"kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-kube-api-access-cgjj5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:31.901821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.901771    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:31.901821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.901781    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:31.901821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:31.901791    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22dede67-e6f7-4577-a8c6-8c4c671814d1-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:32.135413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:32.122034    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"]
Oct 30 21:50:32.145725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:32.145689    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-j8r6f"]
Oct 30 21:50:33.080785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:33.080756    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" path="/var/lib/kubelet/pods/22dede67-e6f7-4577-a8c6-8c4c671814d1/volumes"
Oct 30 21:50:34.617178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:34.617143    4918 scope.go:122] "RemoveContainer" containerID="8975553e65d2eb206068f1db3e675211f97ffa9674c04f9212eae997a99171d8"
Oct 30 21:50:34.629647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:34.629617    4918 scope.go:122] "RemoveContainer" containerID="eebd3a3b9cef64b769e8ff0bf87364da442478227c31fd8940dfd46033b900e1"
Oct 30 21:50:36.683820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:36.683794    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"
Oct 30 21:50:44.653957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.653899    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"]
Oct 30 21:50:44.655334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.655309    4918 cpu_manager.go:414] "containerMap: removing container" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerName="istio-proxy"
Oct 30 21:50:44.655334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.655331    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerName="istio-proxy"
Oct 30 21:50:44.655641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.655624    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dede67-e6f7-4577-a8c6-8c4c671814d1" containerName="istio-proxy"
Oct 30 21:50:44.678437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.678408    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"]
Oct 30 21:50:44.678628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.678537    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:44.684827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.684801    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e8d315-serving-cert\""
Oct 30 21:50:44.843494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.829253    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-maas-api-tls\") pod \"maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:44.843494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.829374    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-metrics-certs\") pod \"maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:44.843494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.829526    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zph4w\" (UniqueName: \"kubernetes.io/projected/9d50305c-28b7-42d5-bda9-3dd6d0c13545-kube-api-access-zph4w\") pod \"maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:44.843494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.829848    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d50305c-28b7-42d5-bda9-3dd6d0c13545-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:44.931207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.931126    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d50305c-28b7-42d5-bda9-3dd6d0c13545-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:44.931207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.931195    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-maas-api-tls\") pod \"maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:44.931511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.931252    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-metrics-certs\") pod \"maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:44.931511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.931315    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zph4w\" (UniqueName: \"kubernetes.io/projected/9d50305c-28b7-42d5-bda9-3dd6d0c13545-kube-api-access-zph4w\") pod \"maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:44.931799 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.931775    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d50305c-28b7-42d5-bda9-3dd6d0c13545-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:44.933917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.933892    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-metrics-certs\") pod \"maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:44.934026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.933989    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-maas-api-tls\") pod \"maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:44.940212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:44.940189    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zph4w\" (UniqueName: \"kubernetes.io/projected/9d50305c-28b7-42d5-bda9-3dd6d0c13545-kube-api-access-zph4w\") pod \"maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:45.008021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:45.007993    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:45.413075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:45.410956    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"]
Oct 30 21:50:45.413075 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:50:45.412919    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d50305c_28b7_42d5_bda9_3dd6d0c13545.slice/crio-0d0484a762837c979b38b55f97f39bf18a83da52ec14f78473ab1fe020743592 WatchSource:0}: Error finding container 0d0484a762837c979b38b55f97f39bf18a83da52ec14f78473ab1fe020743592: Status 404 returned error can't find the container with id 0d0484a762837c979b38b55f97f39bf18a83da52ec14f78473ab1fe020743592
Oct 30 21:50:45.892416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:45.892355    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s" event={"ID":"9d50305c-28b7-42d5-bda9-3dd6d0c13545","Type":"ContainerStarted","Data":"0d0484a762837c979b38b55f97f39bf18a83da52ec14f78473ab1fe020743592"}
Oct 30 21:50:46.904999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:46.904959    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s" event={"ID":"9d50305c-28b7-42d5-bda9-3dd6d0c13545","Type":"ContainerStarted","Data":"30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a"}
Oct 30 21:50:46.905475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:46.905033    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:46.926207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:46.926151    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s" podStartSLOduration=2.386978598 podStartE2EDuration="2.92613631s" totalImagesPullingTime="539.157712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:46.923911978 +0000 UTC m=+2959.970471145" watchObservedRunningTime="2026-10-30 21:50:46.92613631 +0000 UTC m=+2959.972695455"
Oct 30 21:50:52.937984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:50:52.937950    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:50:59.162154 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:50:59.162119    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:51:12.093054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:12.093008    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"]
Oct 30 21:51:12.104117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:12.104078    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5811b415-schj7"]
Oct 30 21:51:13.082617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:13.082587    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13caba9-8805-41c5-b3c4-f1053933c608" path="/var/lib/kubelet/pods/c13caba9-8805-41c5-b3c4-f1053933c608/volumes"
Oct 30 21:51:22.026533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:22.026498    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"]
Oct 30 21:51:22.031673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:22.031641    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17a9816d-nv7tr"]
Oct 30 21:51:23.082719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:23.082684    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="687e7406-3b53-4568-9cdd-2ba41287568a" path="/var/lib/kubelet/pods/687e7406-3b53-4568-9cdd-2ba41287568a/volumes"
Oct 30 21:51:26.029227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:26.029194    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"]
Oct 30 21:51:26.033949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:26.033925    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef07f59b-h6sj4"]
Oct 30 21:51:27.029955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:27.029920    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"]
Oct 30 21:51:27.034596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:27.034570    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9232cc20-jxcr7"]
Oct 30 21:51:27.081511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:27.081482    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08cf8023-9260-4e4a-9aa3-c801e7a6f85b" path="/var/lib/kubelet/pods/08cf8023-9260-4e4a-9aa3-c801e7a6f85b/volumes"
Oct 30 21:51:27.082021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:27.082005    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74c7e5b-34f0-480e-8f2a-63b406788ed5" path="/var/lib/kubelet/pods/f74c7e5b-34f0-480e-8f2a-63b406788ed5/volumes"
Oct 30 21:51:29.164612 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:51:29.164574    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:51:31.124304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:31.124271    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:51:31.126262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:31.126238    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:51:34.688705 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:34.688673    4918 scope.go:122] "RemoveContainer" containerID="91c1998096c198a8cf8674c0e243b76607b8d18f61ab2cdde1bacd884ac4e7cc"
Oct 30 21:51:34.700597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:34.700568    4918 scope.go:122] "RemoveContainer" containerID="52d6bf9175a21b874c1e002307ec178db9b4da4dec117f2fe354712f00ec9125"
Oct 30 21:51:35.027881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:35.027840    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"]
Oct 30 21:51:35.032581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:35.032553    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92eb4bfd-p7nfr"]
Oct 30 21:51:35.082689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:35.082661    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59" path="/var/lib/kubelet/pods/fdf8f8a1-f957-4f2c-9b3e-0fe68f0e9b59/volumes"
Oct 30 21:51:46.026168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:46.026081    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"]
Oct 30 21:51:46.031725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:46.031691    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdb76d7c-6qv9q"]
Oct 30 21:51:47.082194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:51:47.082163    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f75d56-ce02-465b-8746-ab34b97bfd66" path="/var/lib/kubelet/pods/a0f75d56-ce02-465b-8746-ab34b97bfd66/volumes"
Oct 30 21:51:59.162557 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:51:59.162523    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:52:05.341332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.341299    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"]
Oct 30 21:52:05.346548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.346520    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:05.354793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.354766    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"]
Oct 30 21:52:05.354936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.354857    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:52:05.354936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.354927    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:52:05.459611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.459578    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98394fc5-jczws\" (UID: \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:05.459848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.459825    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7mk\" (UniqueName: \"kubernetes.io/projected/010ec348-3f3c-46f5-ba7f-55ebbf70029b-kube-api-access-sj7mk\") pod \"maas-api-revoke-sub-98394fc5-jczws\" (UID: \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:05.459988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.459973    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98394fc5-jczws\" (UID: \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:05.562368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.562247    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98394fc5-jczws\" (UID: \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:05.562709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.562690    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7mk\" (UniqueName: \"kubernetes.io/projected/010ec348-3f3c-46f5-ba7f-55ebbf70029b-kube-api-access-sj7mk\") pod \"maas-api-revoke-sub-98394fc5-jczws\" (UID: \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:05.562874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.562854    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98394fc5-jczws\" (UID: \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:05.563113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.563007    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98394fc5-jczws\" (UID: \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:05.565668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.565646    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98394fc5-jczws\" (UID: \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:05.571930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.571910    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7mk\" (UniqueName: \"kubernetes.io/projected/010ec348-3f3c-46f5-ba7f-55ebbf70029b-kube-api-access-sj7mk\") pod \"maas-api-revoke-sub-98394fc5-jczws\" (UID: \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:05.672142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.672068    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:05.973020 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:52:05.972985    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010ec348_3f3c_46f5_ba7f_55ebbf70029b.slice/crio-27f84934acd4b88beb3a102576e8489f3e3f5db5f713eda22dc70acdf98007ef WatchSource:0}: Error finding container 27f84934acd4b88beb3a102576e8489f3e3f5db5f713eda22dc70acdf98007ef: Status 404 returned error can't find the container with id 27f84934acd4b88beb3a102576e8489f3e3f5db5f713eda22dc70acdf98007ef
Oct 30 21:52:05.973195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:05.973072    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"]
Oct 30 21:52:06.127679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:06.127640    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws" event={"ID":"010ec348-3f3c-46f5-ba7f-55ebbf70029b","Type":"ContainerStarted","Data":"27f84934acd4b88beb3a102576e8489f3e3f5db5f713eda22dc70acdf98007ef"}
Oct 30 21:52:07.028310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:07.028281    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"]
Oct 30 21:52:07.028823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:07.028510    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x" podUID="85acd5b8-9974-4948-9713-ea28737f0d8c" containerName="istio-proxy" containerID="cri-o://4231d826ce284be8b4e1eebb48cffcaaa6714b37996e07dd0343f2725c8b6fc4" gracePeriod=30
Oct 30 21:52:07.138462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:07.138429    4918 generic.go:356] "Generic (PLEG): container finished" podID="010ec348-3f3c-46f5-ba7f-55ebbf70029b" containerID="e04d1df95940f40b66568bc5a56841d8ce5439098f0d5601e1d34e1dae8283e8" exitCode=0
Oct 30 21:52:07.138623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:07.138497    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws" event={"ID":"010ec348-3f3c-46f5-ba7f-55ebbf70029b","Type":"ContainerDied","Data":"e04d1df95940f40b66568bc5a56841d8ce5439098f0d5601e1d34e1dae8283e8"}
Oct 30 21:52:08.326586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:08.326555    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:08.447585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:08.447504    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-openshift-ca\") pod \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\" (UID: \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\") "
Oct 30 21:52:08.447747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:08.447597    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010ec348-3f3c-46f5-ba7f-55ebbf70029b-kube-api-access-sj7mk\" (UniqueName: \"kubernetes.io/projected/010ec348-3f3c-46f5-ba7f-55ebbf70029b-kube-api-access-sj7mk\") pod \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\" (UID: \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\") "
Oct 30 21:52:08.447805 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:08.447740    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-xks-ca\") pod \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\" (UID: \"010ec348-3f3c-46f5-ba7f-55ebbf70029b\") "
Oct 30 21:52:08.447879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:08.447853    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-openshift-ca" pod "010ec348-3f3c-46f5-ba7f-55ebbf70029b" (UID: "010ec348-3f3c-46f5-ba7f-55ebbf70029b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:08.448506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:08.448486    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:08.450017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:08.449992    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-xks-ca" pod "010ec348-3f3c-46f5-ba7f-55ebbf70029b" (UID: "010ec348-3f3c-46f5-ba7f-55ebbf70029b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:52:08.450117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:08.450055    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010ec348-3f3c-46f5-ba7f-55ebbf70029b-kube-api-access-sj7mk" pod "010ec348-3f3c-46f5-ba7f-55ebbf70029b" (UID: "010ec348-3f3c-46f5-ba7f-55ebbf70029b"). InnerVolumeSpecName "kube-api-access-sj7mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:08.574124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:08.574089    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj7mk\" (UniqueName: \"kubernetes.io/projected/010ec348-3f3c-46f5-ba7f-55ebbf70029b-kube-api-access-sj7mk\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:08.574427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:08.574404    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/010ec348-3f3c-46f5-ba7f-55ebbf70029b-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:09.158720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:09.158692    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"
Oct 30 21:52:09.158720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:09.158717    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws" event={"ID":"010ec348-3f3c-46f5-ba7f-55ebbf70029b","Type":"ContainerDied","Data":"27f84934acd4b88beb3a102576e8489f3e3f5db5f713eda22dc70acdf98007ef"}
Oct 30 21:52:09.158936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:09.158760    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27f84934acd4b88beb3a102576e8489f3e3f5db5f713eda22dc70acdf98007ef"
Oct 30 21:52:12.234668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.234418    4918 generic.go:356] "Generic (PLEG): container finished" podID="85acd5b8-9974-4948-9713-ea28737f0d8c" containerID="4231d826ce284be8b4e1eebb48cffcaaa6714b37996e07dd0343f2725c8b6fc4" exitCode=0
Oct 30 21:52:12.234668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.234586    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x" event={"ID":"85acd5b8-9974-4948-9713-ea28737f0d8c","Type":"ContainerDied","Data":"4231d826ce284be8b4e1eebb48cffcaaa6714b37996e07dd0343f2725c8b6fc4"}
Oct 30 21:52:12.406185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.406158    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:52:12.580275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.580194    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-data\") pod \"85acd5b8-9974-4948-9713-ea28737f0d8c\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") "
Oct 30 21:52:12.580275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.580232    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-token\" (UniqueName: \"kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-token\") pod \"85acd5b8-9974-4948-9713-ea28737f0d8c\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") "
Oct 30 21:52:12.580275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.580249    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-certs\") pod \"85acd5b8-9974-4948-9713-ea28737f0d8c\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") "
Oct 30 21:52:12.580275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.580276    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-kube-api-access-j9qz4\" (UniqueName: \"kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-kube-api-access-j9qz4\") pod \"85acd5b8-9974-4948-9713-ea28737f0d8c\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") "
Oct 30 21:52:12.580682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.580497    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-podinfo\") pod \"85acd5b8-9974-4948-9713-ea28737f0d8c\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") "
Oct 30 21:52:12.580682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.580517    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-credential-socket\") pod \"85acd5b8-9974-4948-9713-ea28737f0d8c\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") "
Oct 30 21:52:12.580682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.580561    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85acd5b8-9974-4948-9713-ea28737f0d8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85acd5b8-9974-4948-9713-ea28737f0d8c-istiod-ca-cert\") pod \"85acd5b8-9974-4948-9713-ea28737f0d8c\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") "
Oct 30 21:52:12.580682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.580601    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-data" pod "85acd5b8-9974-4948-9713-ea28737f0d8c" (UID: "85acd5b8-9974-4948-9713-ea28737f0d8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:12.580682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.580621    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-certs" pod "85acd5b8-9974-4948-9713-ea28737f0d8c" (UID: "85acd5b8-9974-4948-9713-ea28737f0d8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:12.580682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.580654    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-envoy\") pod \"85acd5b8-9974-4948-9713-ea28737f0d8c\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") "
Oct 30 21:52:12.581025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.580746    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-socket\") pod \"85acd5b8-9974-4948-9713-ea28737f0d8c\" (UID: \"85acd5b8-9974-4948-9713-ea28737f0d8c\") "
Oct 30 21:52:12.581088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.581037    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-socket" pod "85acd5b8-9974-4948-9713-ea28737f0d8c" (UID: "85acd5b8-9974-4948-9713-ea28737f0d8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:12.581088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.581069    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-credential-socket" pod "85acd5b8-9974-4948-9713-ea28737f0d8c" (UID: "85acd5b8-9974-4948-9713-ea28737f0d8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:12.581088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.581069    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85acd5b8-9974-4948-9713-ea28737f0d8c-istiod-ca-cert" pod "85acd5b8-9974-4948-9713-ea28737f0d8c" (UID: "85acd5b8-9974-4948-9713-ea28737f0d8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:12.581671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.581650    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:12.581778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.581673    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:12.581778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.581686    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:12.581778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.581700    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:12.581778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.581712    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85acd5b8-9974-4948-9713-ea28737f0d8c-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:12.582907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.582874    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-token" pod "85acd5b8-9974-4948-9713-ea28737f0d8c" (UID: "85acd5b8-9974-4948-9713-ea28737f0d8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:12.583049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.583020    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-podinfo" pod "85acd5b8-9974-4948-9713-ea28737f0d8c" (UID: "85acd5b8-9974-4948-9713-ea28737f0d8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:52:12.583469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.583381    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-kube-api-access-j9qz4" pod "85acd5b8-9974-4948-9713-ea28737f0d8c" (UID: "85acd5b8-9974-4948-9713-ea28737f0d8c"). InnerVolumeSpecName "kube-api-access-j9qz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:12.583580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.583465    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-envoy" pod "85acd5b8-9974-4948-9713-ea28737f0d8c" (UID: "85acd5b8-9974-4948-9713-ea28737f0d8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:12.683143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.683106    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:12.683143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.683141    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:12.683347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.683151    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:12.683347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:12.683161    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9qz4\" (UniqueName: \"kubernetes.io/projected/85acd5b8-9974-4948-9713-ea28737f0d8c-kube-api-access-j9qz4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:13.244849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:13.244816    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"
Oct 30 21:52:13.244849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:13.244834    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x" event={"ID":"85acd5b8-9974-4948-9713-ea28737f0d8c","Type":"ContainerDied","Data":"ebf2452d2f3c1fd67e7137ee21aa0d4f98e2373ebc91d1cc5eadf9c5a71fac79"}
Oct 30 21:52:13.245360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:13.244890    4918 scope.go:122] "RemoveContainer" containerID="4231d826ce284be8b4e1eebb48cffcaaa6714b37996e07dd0343f2725c8b6fc4"
Oct 30 21:52:13.269033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:13.269011    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"]
Oct 30 21:52:13.274182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:13.274158    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-j526x"]
Oct 30 21:52:15.079902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:15.079868    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85acd5b8-9974-4948-9713-ea28737f0d8c" path="/var/lib/kubelet/pods/85acd5b8-9974-4948-9713-ea28737f0d8c/volumes"
Oct 30 21:52:16.618139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.618105    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"]
Oct 30 21:52:16.619034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.619016    4918 cpu_manager.go:414] "containerMap: removing container" podUID="010ec348-3f3c-46f5-ba7f-55ebbf70029b" containerName="revoke-keys"
Oct 30 21:52:16.619085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.619036    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010ec348-3f3c-46f5-ba7f-55ebbf70029b" containerName="revoke-keys"
Oct 30 21:52:16.619085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.619068    4918 cpu_manager.go:414] "containerMap: removing container" podUID="85acd5b8-9974-4948-9713-ea28737f0d8c" containerName="istio-proxy"
Oct 30 21:52:16.619085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.619072    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85acd5b8-9974-4948-9713-ea28737f0d8c" containerName="istio-proxy"
Oct 30 21:52:16.619236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.619226    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="85acd5b8-9974-4948-9713-ea28737f0d8c" containerName="istio-proxy"
Oct 30 21:52:16.619274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.619245    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="010ec348-3f3c-46f5-ba7f-55ebbf70029b" containerName="revoke-keys"
Oct 30 21:52:16.625365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.625338    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:16.630431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.630376    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:52:16.630571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.630498    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:52:16.631321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.631300    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"]
Oct 30 21:52:16.730084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.730053    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwfw\" (UniqueName: \"kubernetes.io/projected/fc44b1a9-3531-4107-8960-d4c7fbc51757-kube-api-access-vhwfw\") pod \"maas-api-revoke-sub-c2bc375c-2xr8m\" (UID: \"fc44b1a9-3531-4107-8960-d4c7fbc51757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:16.730261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.730098    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2bc375c-2xr8m\" (UID: \"fc44b1a9-3531-4107-8960-d4c7fbc51757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:16.730377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.730356    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2bc375c-2xr8m\" (UID: \"fc44b1a9-3531-4107-8960-d4c7fbc51757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:16.832329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.832299    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwfw\" (UniqueName: \"kubernetes.io/projected/fc44b1a9-3531-4107-8960-d4c7fbc51757-kube-api-access-vhwfw\") pod \"maas-api-revoke-sub-c2bc375c-2xr8m\" (UID: \"fc44b1a9-3531-4107-8960-d4c7fbc51757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:16.832522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.832336    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2bc375c-2xr8m\" (UID: \"fc44b1a9-3531-4107-8960-d4c7fbc51757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:16.832522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.832476    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2bc375c-2xr8m\" (UID: \"fc44b1a9-3531-4107-8960-d4c7fbc51757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:16.833173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.833091    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2bc375c-2xr8m\" (UID: \"fc44b1a9-3531-4107-8960-d4c7fbc51757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:16.834872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.834853    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2bc375c-2xr8m\" (UID: \"fc44b1a9-3531-4107-8960-d4c7fbc51757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:16.840750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.840727    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwfw\" (UniqueName: \"kubernetes.io/projected/fc44b1a9-3531-4107-8960-d4c7fbc51757-kube-api-access-vhwfw\") pod \"maas-api-revoke-sub-c2bc375c-2xr8m\" (UID: \"fc44b1a9-3531-4107-8960-d4c7fbc51757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:16.944007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:16.943929    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:17.234605 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:52:17.234571    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc44b1a9_3531_4107_8960_d4c7fbc51757.slice/crio-f6fb81f13f298dc58ad217612fad852e1912a67b2ee0afc99345de8700a3992a WatchSource:0}: Error finding container f6fb81f13f298dc58ad217612fad852e1912a67b2ee0afc99345de8700a3992a: Status 404 returned error can't find the container with id f6fb81f13f298dc58ad217612fad852e1912a67b2ee0afc99345de8700a3992a
Oct 30 21:52:17.236141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:17.236108    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"]
Oct 30 21:52:17.382829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:17.382792    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m" event={"ID":"fc44b1a9-3531-4107-8960-d4c7fbc51757","Type":"ContainerStarted","Data":"f6fb81f13f298dc58ad217612fad852e1912a67b2ee0afc99345de8700a3992a"}
Oct 30 21:52:18.393885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:18.393836    4918 generic.go:356] "Generic (PLEG): container finished" podID="fc44b1a9-3531-4107-8960-d4c7fbc51757" containerID="df22001c9b868b35d9190a173fc7d0f82888e8ed028e5578e3d5a028d17653e5" exitCode=0
Oct 30 21:52:18.394291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:18.393893    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m" event={"ID":"fc44b1a9-3531-4107-8960-d4c7fbc51757","Type":"ContainerDied","Data":"df22001c9b868b35d9190a173fc7d0f82888e8ed028e5578e3d5a028d17653e5"}
Oct 30 21:52:19.647910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:19.647885    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:19.712345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:19.712269    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-openshift-ca\") pod \"fc44b1a9-3531-4107-8960-d4c7fbc51757\" (UID: \"fc44b1a9-3531-4107-8960-d4c7fbc51757\") "
Oct 30 21:52:19.712514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:19.712382    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-xks-ca\") pod \"fc44b1a9-3531-4107-8960-d4c7fbc51757\" (UID: \"fc44b1a9-3531-4107-8960-d4c7fbc51757\") "
Oct 30 21:52:19.712620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:19.712603    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc44b1a9-3531-4107-8960-d4c7fbc51757-kube-api-access-vhwfw\" (UniqueName: \"kubernetes.io/projected/fc44b1a9-3531-4107-8960-d4c7fbc51757-kube-api-access-vhwfw\") pod \"fc44b1a9-3531-4107-8960-d4c7fbc51757\" (UID: \"fc44b1a9-3531-4107-8960-d4c7fbc51757\") "
Oct 30 21:52:19.712665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:19.712630    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-openshift-ca" pod "fc44b1a9-3531-4107-8960-d4c7fbc51757" (UID: "fc44b1a9-3531-4107-8960-d4c7fbc51757"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:19.713188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:19.713111    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:19.714820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:19.714791    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-xks-ca" pod "fc44b1a9-3531-4107-8960-d4c7fbc51757" (UID: "fc44b1a9-3531-4107-8960-d4c7fbc51757"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:52:19.715175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:19.715157    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc44b1a9-3531-4107-8960-d4c7fbc51757-kube-api-access-vhwfw" pod "fc44b1a9-3531-4107-8960-d4c7fbc51757" (UID: "fc44b1a9-3531-4107-8960-d4c7fbc51757"). InnerVolumeSpecName "kube-api-access-vhwfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:19.815211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:19.815177    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhwfw\" (UniqueName: \"kubernetes.io/projected/fc44b1a9-3531-4107-8960-d4c7fbc51757-kube-api-access-vhwfw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:19.815211 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:19.815208    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc44b1a9-3531-4107-8960-d4c7fbc51757-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:20.414264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:20.414238    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"
Oct 30 21:52:20.414474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:20.414271    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m" event={"ID":"fc44b1a9-3531-4107-8960-d4c7fbc51757","Type":"ContainerDied","Data":"f6fb81f13f298dc58ad217612fad852e1912a67b2ee0afc99345de8700a3992a"}
Oct 30 21:52:20.414474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:20.414315    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6fb81f13f298dc58ad217612fad852e1912a67b2ee0afc99345de8700a3992a"
Oct 30 21:52:27.559611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.559577    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"]
Oct 30 21:52:27.560444 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.560429    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fc44b1a9-3531-4107-8960-d4c7fbc51757" containerName="revoke-keys"
Oct 30 21:52:27.560507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.560446    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc44b1a9-3531-4107-8960-d4c7fbc51757" containerName="revoke-keys"
Oct 30 21:52:27.560651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.560641    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc44b1a9-3531-4107-8960-d4c7fbc51757" containerName="revoke-keys"
Oct 30 21:52:27.564210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.564186    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:27.569280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.569255    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:52:27.569558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.569539    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:52:27.575333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.575311    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"]
Oct 30 21:52:27.710736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.710701    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m57g\" (UniqueName: \"kubernetes.io/projected/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-kube-api-access-4m57g\") pod \"maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc\" (UID: \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:27.710961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.710946    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc\" (UID: \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:27.711035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.710971    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc\" (UID: \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:27.812150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.812056    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc\" (UID: \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:27.812150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.812101    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc\" (UID: \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:27.812365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.812293    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m57g\" (UniqueName: \"kubernetes.io/projected/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-kube-api-access-4m57g\") pod \"maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc\" (UID: \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:27.812755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.812730    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc\" (UID: \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:27.814812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.814793    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc\" (UID: \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:27.820506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.820486    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m57g\" (UniqueName: \"kubernetes.io/projected/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-kube-api-access-4m57g\") pod \"maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc\" (UID: \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:27.881874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:27.881848    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:28.169607 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:52:28.169576    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4fdeec4_9e03_4dad_8dbd_c2a8c05f2355.slice/crio-49efabd56f273f10fcf398237ba0ed8c2eee0f856a7f2c748622175ffbc2ce5c WatchSource:0}: Error finding container 49efabd56f273f10fcf398237ba0ed8c2eee0f856a7f2c748622175ffbc2ce5c: Status 404 returned error can't find the container with id 49efabd56f273f10fcf398237ba0ed8c2eee0f856a7f2c748622175ffbc2ce5c
Oct 30 21:52:28.170302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:28.170279    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"]
Oct 30 21:52:28.556413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:28.555645    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc" event={"ID":"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355","Type":"ContainerStarted","Data":"49efabd56f273f10fcf398237ba0ed8c2eee0f856a7f2c748622175ffbc2ce5c"}
Oct 30 21:52:29.161963 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:52:29.161928    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:52:29.566931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:29.566888    4918 generic.go:356] "Generic (PLEG): container finished" podID="a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355" containerID="960472a07ddb7f982786003e6ef9c964e68b6e88bce2778d67da93fd9a4aec60" exitCode=0
Oct 30 21:52:29.567131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:29.566952    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc" event={"ID":"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355","Type":"ContainerDied","Data":"960472a07ddb7f982786003e6ef9c964e68b6e88bce2778d67da93fd9a4aec60"}
Oct 30 21:52:30.755354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:30.755331    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:30.801680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:30.801653    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-kube-api-access-4m57g\" (UniqueName: \"kubernetes.io/projected/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-kube-api-access-4m57g\") pod \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\" (UID: \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\") "
Oct 30 21:52:30.801859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:30.801713    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-openshift-ca\") pod \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\" (UID: \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\") "
Oct 30 21:52:30.801932 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:30.801853    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-xks-ca\") pod \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\" (UID: \"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355\") "
Oct 30 21:52:30.802101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:30.802079    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-openshift-ca" pod "a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355" (UID: "a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:30.802827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:30.802796    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:30.804108 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:30.804086    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-xks-ca" pod "a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355" (UID: "a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:52:30.804189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:30.804175    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-kube-api-access-4m57g" pod "a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355" (UID: "a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355"). InnerVolumeSpecName "kube-api-access-4m57g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:30.906041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:30.905883    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m57g\" (UniqueName: \"kubernetes.io/projected/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-kube-api-access-4m57g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:30.906041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:30.905924    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:31.591499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:31.591458    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc" event={"ID":"a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355","Type":"ContainerDied","Data":"49efabd56f273f10fcf398237ba0ed8c2eee0f856a7f2c748622175ffbc2ce5c"}
Oct 30 21:52:31.591691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:31.591544    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49efabd56f273f10fcf398237ba0ed8c2eee0f856a7f2c748622175ffbc2ce5c"
Oct 30 21:52:31.591691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:31.591607    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"
Oct 30 21:52:34.742234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:34.742205    4918 scope.go:122] "RemoveContainer" containerID="515e52bf4982eed9b4cdd012248ff8ddb62544b20003d2290cac7f8988aa7026"
Oct 30 21:52:34.754399 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:34.754362    4918 scope.go:122] "RemoveContainer" containerID="a95124a4188ba95cf4aab23f1234fa58d53180d5cfc00a56b13cb652ba6fe7b6"
Oct 30 21:52:34.766769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:34.766739    4918 scope.go:122] "RemoveContainer" containerID="349a191a77241ddab187b646e28dcc2f03bacb2d55ad249e26cfafdaac535191"
Oct 30 21:52:34.776763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:34.776743    4918 scope.go:122] "RemoveContainer" containerID="bfddf001ae2a0912506c402aa103a328a2d45336e12174bf41a9ec549b0cdc10"
Oct 30 21:52:34.786674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:34.786655    4918 scope.go:122] "RemoveContainer" containerID="f70e06a5cf82e46e5653f0b1e001af141339b20aed6f70df061db10bf698004a"
Oct 30 21:52:47.517657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.517622    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"]
Oct 30 21:52:47.518618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.518601    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355" containerName="revoke-keys"
Oct 30 21:52:47.518618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.518619    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355" containerName="revoke-keys"
Oct 30 21:52:47.518767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.518757    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355" containerName="revoke-keys"
Oct 30 21:52:47.525374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.525341    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:47.529511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.529483    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:52:47.530416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.530378    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"]
Oct 30 21:52:47.531081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.531058    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:52:47.661804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.661764    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htmqz\" (UniqueName: \"kubernetes.io/projected/6c6ee490-7031-4a6d-a794-04ec2d0a360e-kube-api-access-htmqz\") pod \"maas-api-revoke-sub-9f831668-gdhh4\" (UID: \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:47.662067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.662048    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f831668-gdhh4\" (UID: \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:47.662193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.662175    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f831668-gdhh4\" (UID: \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:47.764182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.764146    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f831668-gdhh4\" (UID: \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:47.764464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.764266    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htmqz\" (UniqueName: \"kubernetes.io/projected/6c6ee490-7031-4a6d-a794-04ec2d0a360e-kube-api-access-htmqz\") pod \"maas-api-revoke-sub-9f831668-gdhh4\" (UID: \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:47.764555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.764489    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f831668-gdhh4\" (UID: \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:47.765314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.765268    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f831668-gdhh4\" (UID: \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:47.766977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.766948    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f831668-gdhh4\" (UID: \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:47.773662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.773611    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htmqz\" (UniqueName: \"kubernetes.io/projected/6c6ee490-7031-4a6d-a794-04ec2d0a360e-kube-api-access-htmqz\") pod \"maas-api-revoke-sub-9f831668-gdhh4\" (UID: \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:47.852418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:47.845620    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:48.155794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:48.155767    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"]
Oct 30 21:52:48.156120 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:52:48.156099    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6ee490_7031_4a6d_a794_04ec2d0a360e.slice/crio-b4d7df313fc1172fa8e4d55f1ad0c25d9fe79845285836498abcb4b1a2991f8f WatchSource:0}: Error finding container b4d7df313fc1172fa8e4d55f1ad0c25d9fe79845285836498abcb4b1a2991f8f: Status 404 returned error can't find the container with id b4d7df313fc1172fa8e4d55f1ad0c25d9fe79845285836498abcb4b1a2991f8f
Oct 30 21:52:48.744814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:48.744775    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4" event={"ID":"6c6ee490-7031-4a6d-a794-04ec2d0a360e","Type":"ContainerStarted","Data":"09d873a5a207551131bcc3b26e103458cdef166fd0ca5449e723aed22345aa9f"}
Oct 30 21:52:48.745274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:48.744841    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4" event={"ID":"6c6ee490-7031-4a6d-a794-04ec2d0a360e","Type":"ContainerStarted","Data":"b4d7df313fc1172fa8e4d55f1ad0c25d9fe79845285836498abcb4b1a2991f8f"}
Oct 30 21:52:48.764955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:48.764908    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4" podStartSLOduration=1.7648953889999999 podStartE2EDuration="1.764895389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:48.763586405 +0000 UTC m=+3081.810145570" watchObservedRunningTime="2026-10-30 21:52:48.764895389 +0000 UTC m=+3081.811454542"
Oct 30 21:52:49.754987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:49.754953    4918 generic.go:356] "Generic (PLEG): container finished" podID="6c6ee490-7031-4a6d-a794-04ec2d0a360e" containerID="09d873a5a207551131bcc3b26e103458cdef166fd0ca5449e723aed22345aa9f" exitCode=0
Oct 30 21:52:49.755423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:49.755040    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4" event={"ID":"6c6ee490-7031-4a6d-a794-04ec2d0a360e","Type":"ContainerDied","Data":"09d873a5a207551131bcc3b26e103458cdef166fd0ca5449e723aed22345aa9f"}
Oct 30 21:52:50.960097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:50.960074    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:51.054562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:51.054467    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6ee490-7031-4a6d-a794-04ec2d0a360e-kube-api-access-htmqz\" (UniqueName: \"kubernetes.io/projected/6c6ee490-7031-4a6d-a794-04ec2d0a360e-kube-api-access-htmqz\") pod \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\" (UID: \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\") "
Oct 30 21:52:51.054750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:51.054591    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-xks-ca\") pod \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\" (UID: \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\") "
Oct 30 21:52:51.054935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:51.054913    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-openshift-ca\") pod \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\" (UID: \"6c6ee490-7031-4a6d-a794-04ec2d0a360e\") "
Oct 30 21:52:51.055348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:51.055283    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-openshift-ca" pod "6c6ee490-7031-4a6d-a794-04ec2d0a360e" (UID: "6c6ee490-7031-4a6d-a794-04ec2d0a360e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:51.055752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:51.055731    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:51.056783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:51.056754    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6ee490-7031-4a6d-a794-04ec2d0a360e-kube-api-access-htmqz" pod "6c6ee490-7031-4a6d-a794-04ec2d0a360e" (UID: "6c6ee490-7031-4a6d-a794-04ec2d0a360e"). InnerVolumeSpecName "kube-api-access-htmqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:51.056887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:51.056829    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-xks-ca" pod "6c6ee490-7031-4a6d-a794-04ec2d0a360e" (UID: "6c6ee490-7031-4a6d-a794-04ec2d0a360e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:52:51.157738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:51.157706    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htmqz\" (UniqueName: \"kubernetes.io/projected/6c6ee490-7031-4a6d-a794-04ec2d0a360e-kube-api-access-htmqz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:51.157738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:51.157736    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6ee490-7031-4a6d-a794-04ec2d0a360e-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:51.774326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:51.774292    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"
Oct 30 21:52:51.774540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:51.774295    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4" event={"ID":"6c6ee490-7031-4a6d-a794-04ec2d0a360e","Type":"ContainerDied","Data":"b4d7df313fc1172fa8e4d55f1ad0c25d9fe79845285836498abcb4b1a2991f8f"}
Oct 30 21:52:51.774540 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:51.774463    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4d7df313fc1172fa8e4d55f1ad0c25d9fe79845285836498abcb4b1a2991f8f"
Oct 30 21:52:58.006019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:58.005207    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"]
Oct 30 21:52:58.009560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:58.009535    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8431de-jmfrc"]
Oct 30 21:52:59.062843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.062806    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"]
Oct 30 21:52:59.063941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.063919    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6c6ee490-7031-4a6d-a794-04ec2d0a360e" containerName="revoke-keys"
Oct 30 21:52:59.063941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.063942    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6ee490-7031-4a6d-a794-04ec2d0a360e" containerName="revoke-keys"
Oct 30 21:52:59.064242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.064226    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6ee490-7031-4a6d-a794-04ec2d0a360e" containerName="revoke-keys"
Oct 30 21:52:59.068722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.068700    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:52:59.074965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.074935    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:52:59.074965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.074947    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:52:59.085877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.085852    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355" path="/var/lib/kubelet/pods/a4fdeec4-9e03-4dad-8dbd-c2a8c05f2355/volumes"
Oct 30 21:52:59.086291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.086271    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"]
Oct 30 21:52:59.162855 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:52:59.162822    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:52:59.232405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.232356    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9c0bcd-vmkc8\" (UID: \"fe86e610-3b67-487f-9a62-8355e7de6cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:52:59.232585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.232501    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbq6\" (UniqueName: \"kubernetes.io/projected/fe86e610-3b67-487f-9a62-8355e7de6cd1-kube-api-access-fsbq6\") pod \"maas-api-revoke-sub-2a9c0bcd-vmkc8\" (UID: \"fe86e610-3b67-487f-9a62-8355e7de6cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:52:59.232903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.232882    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9c0bcd-vmkc8\" (UID: \"fe86e610-3b67-487f-9a62-8355e7de6cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:52:59.346467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.340636    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9c0bcd-vmkc8\" (UID: \"fe86e610-3b67-487f-9a62-8355e7de6cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:52:59.346467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.340926    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9c0bcd-vmkc8\" (UID: \"fe86e610-3b67-487f-9a62-8355e7de6cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:52:59.346467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.341000    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbq6\" (UniqueName: \"kubernetes.io/projected/fe86e610-3b67-487f-9a62-8355e7de6cd1-kube-api-access-fsbq6\") pod \"maas-api-revoke-sub-2a9c0bcd-vmkc8\" (UID: \"fe86e610-3b67-487f-9a62-8355e7de6cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:52:59.346467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.342305    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9c0bcd-vmkc8\" (UID: \"fe86e610-3b67-487f-9a62-8355e7de6cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:52:59.347958 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.347919    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9c0bcd-vmkc8\" (UID: \"fe86e610-3b67-487f-9a62-8355e7de6cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:52:59.358676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.358649    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbq6\" (UniqueName: \"kubernetes.io/projected/fe86e610-3b67-487f-9a62-8355e7de6cd1-kube-api-access-fsbq6\") pod \"maas-api-revoke-sub-2a9c0bcd-vmkc8\" (UID: \"fe86e610-3b67-487f-9a62-8355e7de6cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:52:59.390549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.390513    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:52:59.677032 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:52:59.676998    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe86e610_3b67_487f_9a62_8355e7de6cd1.slice/crio-d4d50d0595c2b6fb730a14a0f686b4a2f3fcdf49450ec0e8abdd475157dd1637 WatchSource:0}: Error finding container d4d50d0595c2b6fb730a14a0f686b4a2f3fcdf49450ec0e8abdd475157dd1637: Status 404 returned error can't find the container with id d4d50d0595c2b6fb730a14a0f686b4a2f3fcdf49450ec0e8abdd475157dd1637
Oct 30 21:52:59.677183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.677049    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"]
Oct 30 21:52:59.972787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:52:59.972758    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8" event={"ID":"fe86e610-3b67-487f-9a62-8355e7de6cd1","Type":"ContainerStarted","Data":"d4d50d0595c2b6fb730a14a0f686b4a2f3fcdf49450ec0e8abdd475157dd1637"}
Oct 30 21:53:00.984004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:00.983971    4918 generic.go:356] "Generic (PLEG): container finished" podID="fe86e610-3b67-487f-9a62-8355e7de6cd1" containerID="b244d5a58bd4bd666c7fdd964b54e1a54c6b94d36fa59e5537d78c8c726521f8" exitCode=0
Oct 30 21:53:00.984436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:00.984039    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8" event={"ID":"fe86e610-3b67-487f-9a62-8355e7de6cd1","Type":"ContainerDied","Data":"b244d5a58bd4bd666c7fdd964b54e1a54c6b94d36fa59e5537d78c8c726521f8"}
Oct 30 21:53:02.175922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:02.175892    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:53:02.224656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:02.224622    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-xks-ca\") pod \"fe86e610-3b67-487f-9a62-8355e7de6cd1\" (UID: \"fe86e610-3b67-487f-9a62-8355e7de6cd1\") "
Oct 30 21:53:02.224868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:02.224814    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe86e610-3b67-487f-9a62-8355e7de6cd1-kube-api-access-fsbq6\" (UniqueName: \"kubernetes.io/projected/fe86e610-3b67-487f-9a62-8355e7de6cd1-kube-api-access-fsbq6\") pod \"fe86e610-3b67-487f-9a62-8355e7de6cd1\" (UID: \"fe86e610-3b67-487f-9a62-8355e7de6cd1\") "
Oct 30 21:53:02.225085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:02.225061    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-openshift-ca\") pod \"fe86e610-3b67-487f-9a62-8355e7de6cd1\" (UID: \"fe86e610-3b67-487f-9a62-8355e7de6cd1\") "
Oct 30 21:53:02.225558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:02.225529    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-openshift-ca" pod "fe86e610-3b67-487f-9a62-8355e7de6cd1" (UID: "fe86e610-3b67-487f-9a62-8355e7de6cd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:02.225957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:02.225937    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:02.227637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:02.227612    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-xks-ca" pod "fe86e610-3b67-487f-9a62-8355e7de6cd1" (UID: "fe86e610-3b67-487f-9a62-8355e7de6cd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:02.228249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:02.228224    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe86e610-3b67-487f-9a62-8355e7de6cd1-kube-api-access-fsbq6" pod "fe86e610-3b67-487f-9a62-8355e7de6cd1" (UID: "fe86e610-3b67-487f-9a62-8355e7de6cd1"). InnerVolumeSpecName "kube-api-access-fsbq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:02.327862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:02.327831    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsbq6\" (UniqueName: \"kubernetes.io/projected/fe86e610-3b67-487f-9a62-8355e7de6cd1-kube-api-access-fsbq6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:02.327862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:02.327865    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe86e610-3b67-487f-9a62-8355e7de6cd1-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:03.003554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:03.003525    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"
Oct 30 21:53:03.003746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:03.003560    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8" event={"ID":"fe86e610-3b67-487f-9a62-8355e7de6cd1","Type":"ContainerDied","Data":"d4d50d0595c2b6fb730a14a0f686b4a2f3fcdf49450ec0e8abdd475157dd1637"}
Oct 30 21:53:03.003746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:03.003613    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4d50d0595c2b6fb730a14a0f686b4a2f3fcdf49450ec0e8abdd475157dd1637"
Oct 30 21:53:05.274579 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.274542    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"]
Oct 30 21:53:05.275004 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.274807    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerName="istio-proxy" containerID="cri-o://e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1" gracePeriod=30
Oct 30 21:53:05.812038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.811999    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"]
Oct 30 21:53:05.814290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.814266    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fe86e610-3b67-487f-9a62-8355e7de6cd1" containerName="revoke-keys"
Oct 30 21:53:05.814474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.814460    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe86e610-3b67-487f-9a62-8355e7de6cd1" containerName="revoke-keys"
Oct 30 21:53:05.815086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.815065    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe86e610-3b67-487f-9a62-8355e7de6cd1" containerName="revoke-keys"
Oct 30 21:53:05.822129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.821966    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:05.823486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.823462    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"]
Oct 30 21:53:05.828220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.828184    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:53:05.830081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.830036    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:53:05.912975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.912942    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65hl7\" (UniqueName: \"kubernetes.io/projected/9b1ee718-1182-431f-aa3e-64275bceac18-kube-api-access-65hl7\") pod \"maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k\" (UID: \"9b1ee718-1182-431f-aa3e-64275bceac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:05.913161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.913002    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k\" (UID: \"9b1ee718-1182-431f-aa3e-64275bceac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:05.913161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:05.913124    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k\" (UID: \"9b1ee718-1182-431f-aa3e-64275bceac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:06.014676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:06.014644    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65hl7\" (UniqueName: \"kubernetes.io/projected/9b1ee718-1182-431f-aa3e-64275bceac18-kube-api-access-65hl7\") pod \"maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k\" (UID: \"9b1ee718-1182-431f-aa3e-64275bceac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:06.014844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:06.014696    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k\" (UID: \"9b1ee718-1182-431f-aa3e-64275bceac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:06.014844 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:06.014764    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k\" (UID: \"9b1ee718-1182-431f-aa3e-64275bceac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:06.015383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:06.015358    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k\" (UID: \"9b1ee718-1182-431f-aa3e-64275bceac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:06.017221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:06.017200    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k\" (UID: \"9b1ee718-1182-431f-aa3e-64275bceac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:06.023216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:06.023186    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65hl7\" (UniqueName: \"kubernetes.io/projected/9b1ee718-1182-431f-aa3e-64275bceac18-kube-api-access-65hl7\") pod \"maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k\" (UID: \"9b1ee718-1182-431f-aa3e-64275bceac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:06.151515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:06.151426    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:06.457213 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:53:06.457180    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1ee718_1182_431f_aa3e_64275bceac18.slice/crio-e463f1ecd3b4074ced6046aa31872c06b0703147ac8d13adf8e648df2bc5c33d WatchSource:0}: Error finding container e463f1ecd3b4074ced6046aa31872c06b0703147ac8d13adf8e648df2bc5c33d: Status 404 returned error can't find the container with id e463f1ecd3b4074ced6046aa31872c06b0703147ac8d13adf8e648df2bc5c33d
Oct 30 21:53:06.457762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:06.457715    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"]
Oct 30 21:53:07.112927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:07.112846    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k" event={"ID":"9b1ee718-1182-431f-aa3e-64275bceac18","Type":"ContainerStarted","Data":"0959c6513a1dd40b80f00b484e467677060d1b91707a91dcc644b7067c934142"}
Oct 30 21:53:07.112927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:07.112899    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k" event={"ID":"9b1ee718-1182-431f-aa3e-64275bceac18","Type":"ContainerStarted","Data":"e463f1ecd3b4074ced6046aa31872c06b0703147ac8d13adf8e648df2bc5c33d"}
Oct 30 21:53:07.146165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:07.146119    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k" podStartSLOduration=2.146106589 podStartE2EDuration="2.146106589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:07.144001186 +0000 UTC m=+3100.190560332" watchObservedRunningTime="2026-10-30 21:53:07.146106589 +0000 UTC m=+3100.192665733"
Oct 30 21:53:08.123863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:08.123828    4918 generic.go:356] "Generic (PLEG): container finished" podID="9b1ee718-1182-431f-aa3e-64275bceac18" containerID="0959c6513a1dd40b80f00b484e467677060d1b91707a91dcc644b7067c934142" exitCode=0
Oct 30 21:53:08.124270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:08.123913    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k" event={"ID":"9b1ee718-1182-431f-aa3e-64275bceac18","Type":"ContainerDied","Data":"0959c6513a1dd40b80f00b484e467677060d1b91707a91dcc644b7067c934142"}
Oct 30 21:53:09.333829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:09.333794    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:09.346220 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:09.346193    4918 patch_prober.go:29] interesting pod/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:53:09.346329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:09.346259    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:53:09.481478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:09.481449    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1ee718-1182-431f-aa3e-64275bceac18-kube-api-access-65hl7\" (UniqueName: \"kubernetes.io/projected/9b1ee718-1182-431f-aa3e-64275bceac18-kube-api-access-65hl7\") pod \"9b1ee718-1182-431f-aa3e-64275bceac18\" (UID: \"9b1ee718-1182-431f-aa3e-64275bceac18\") "
Oct 30 21:53:09.481720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:09.481704    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-openshift-ca\") pod \"9b1ee718-1182-431f-aa3e-64275bceac18\" (UID: \"9b1ee718-1182-431f-aa3e-64275bceac18\") "
Oct 30 21:53:09.481804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:09.481761    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-xks-ca\") pod \"9b1ee718-1182-431f-aa3e-64275bceac18\" (UID: \"9b1ee718-1182-431f-aa3e-64275bceac18\") "
Oct 30 21:53:09.482132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:09.482095    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-openshift-ca" pod "9b1ee718-1182-431f-aa3e-64275bceac18" (UID: "9b1ee718-1182-431f-aa3e-64275bceac18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:09.482681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:09.482659    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:09.484007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:09.483976    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1ee718-1182-431f-aa3e-64275bceac18-kube-api-access-65hl7" pod "9b1ee718-1182-431f-aa3e-64275bceac18" (UID: "9b1ee718-1182-431f-aa3e-64275bceac18"). InnerVolumeSpecName "kube-api-access-65hl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:09.484114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:09.484064    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-xks-ca" pod "9b1ee718-1182-431f-aa3e-64275bceac18" (UID: "9b1ee718-1182-431f-aa3e-64275bceac18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:09.585464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:09.585278    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65hl7\" (UniqueName: \"kubernetes.io/projected/9b1ee718-1182-431f-aa3e-64275bceac18-kube-api-access-65hl7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:09.585464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:09.585316    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1ee718-1182-431f-aa3e-64275bceac18-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.089924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.089885    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"]
Oct 30 21:53:10.090974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.090948    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9b1ee718-1182-431f-aa3e-64275bceac18" containerName="revoke-keys"
Oct 30 21:53:10.090974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.090972    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1ee718-1182-431f-aa3e-64275bceac18" containerName="revoke-keys"
Oct 30 21:53:10.091140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.091130    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1ee718-1182-431f-aa3e-64275bceac18" containerName="revoke-keys"
Oct 30 21:53:10.095234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.095216    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:10.106550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.106524    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"]
Oct 30 21:53:10.144261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.144227    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"
Oct 30 21:53:10.144474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.144259    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k" event={"ID":"9b1ee718-1182-431f-aa3e-64275bceac18","Type":"ContainerDied","Data":"e463f1ecd3b4074ced6046aa31872c06b0703147ac8d13adf8e648df2bc5c33d"}
Oct 30 21:53:10.144474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.144332    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e463f1ecd3b4074ced6046aa31872c06b0703147ac8d13adf8e648df2bc5c33d"
Oct 30 21:53:10.224591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.224555    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk\" (UID: \"92239e35-8394-4120-b2d2-1344583a8a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:10.224790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.224751    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk\" (UID: \"92239e35-8394-4120-b2d2-1344583a8a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:10.224871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.224791    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f24l6\" (UniqueName: \"kubernetes.io/projected/92239e35-8394-4120-b2d2-1344583a8a4a-kube-api-access-f24l6\") pod \"maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk\" (UID: \"92239e35-8394-4120-b2d2-1344583a8a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:10.327224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.327193    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk\" (UID: \"92239e35-8394-4120-b2d2-1344583a8a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:10.327379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.327295    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk\" (UID: \"92239e35-8394-4120-b2d2-1344583a8a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:10.327379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.327313    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24l6\" (UniqueName: \"kubernetes.io/projected/92239e35-8394-4120-b2d2-1344583a8a4a-kube-api-access-f24l6\") pod \"maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk\" (UID: \"92239e35-8394-4120-b2d2-1344583a8a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:10.327935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.327913    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk\" (UID: \"92239e35-8394-4120-b2d2-1344583a8a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:10.329923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.329902    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk\" (UID: \"92239e35-8394-4120-b2d2-1344583a8a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:10.336922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.336901    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24l6\" (UniqueName: \"kubernetes.io/projected/92239e35-8394-4120-b2d2-1344583a8a4a-kube-api-access-f24l6\") pod \"maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk\" (UID: \"92239e35-8394-4120-b2d2-1344583a8a4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:10.416651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.416623    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:10.595235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.595208    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:53:10.749694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.746159    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-certs\") pod \"6733d22f-3aa0-4a87-949f-14bed46f42ff\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") "
Oct 30 21:53:10.749694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.746470    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-socket\") pod \"6733d22f-3aa0-4a87-949f-14bed46f42ff\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") "
Oct 30 21:53:10.753597 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.753538    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-socket" pod "6733d22f-3aa0-4a87-949f-14bed46f42ff" (UID: "6733d22f-3aa0-4a87-949f-14bed46f42ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:10.753802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.753676    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-certs" pod "6733d22f-3aa0-4a87-949f-14bed46f42ff" (UID: "6733d22f-3aa0-4a87-949f-14bed46f42ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:10.772413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.759307    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"]
Oct 30 21:53:10.786415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.783705    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-token\" (UniqueName: \"kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-token\") pod \"6733d22f-3aa0-4a87-949f-14bed46f42ff\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") "
Oct 30 21:53:10.786415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.783791    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-kube-api-access-k2x9z\" (UniqueName: \"kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-kube-api-access-k2x9z\") pod \"6733d22f-3aa0-4a87-949f-14bed46f42ff\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") "
Oct 30 21:53:10.786415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.783866    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-podinfo\") pod \"6733d22f-3aa0-4a87-949f-14bed46f42ff\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") "
Oct 30 21:53:10.786415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.783955    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-credential-socket\") pod \"6733d22f-3aa0-4a87-949f-14bed46f42ff\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") "
Oct 30 21:53:10.786415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.783984    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6733d22f-3aa0-4a87-949f-14bed46f42ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6733d22f-3aa0-4a87-949f-14bed46f42ff-istiod-ca-cert\") pod \"6733d22f-3aa0-4a87-949f-14bed46f42ff\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") "
Oct 30 21:53:10.786415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.784071    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-envoy\") pod \"6733d22f-3aa0-4a87-949f-14bed46f42ff\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") "
Oct 30 21:53:10.786415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.784147    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-data\") pod \"6733d22f-3aa0-4a87-949f-14bed46f42ff\" (UID: \"6733d22f-3aa0-4a87-949f-14bed46f42ff\") "
Oct 30 21:53:10.786415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.785417    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.786415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.785448    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.807491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.807416    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-podinfo" pod "6733d22f-3aa0-4a87-949f-14bed46f42ff" (UID: "6733d22f-3aa0-4a87-949f-14bed46f42ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:53:10.808572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.807686    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-data" pod "6733d22f-3aa0-4a87-949f-14bed46f42ff" (UID: "6733d22f-3aa0-4a87-949f-14bed46f42ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:10.808572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.807924    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-credential-socket" pod "6733d22f-3aa0-4a87-949f-14bed46f42ff" (UID: "6733d22f-3aa0-4a87-949f-14bed46f42ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:10.808572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.808209    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6733d22f-3aa0-4a87-949f-14bed46f42ff-istiod-ca-cert" pod "6733d22f-3aa0-4a87-949f-14bed46f42ff" (UID: "6733d22f-3aa0-4a87-949f-14bed46f42ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:10.810757 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:53:10.810549    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92239e35_8394_4120_b2d2_1344583a8a4a.slice/crio-888e2be0941810f2e82ec53e590245e6d12c31de9871b831c0bed8a0f957bbb9 WatchSource:0}: Error finding container 888e2be0941810f2e82ec53e590245e6d12c31de9871b831c0bed8a0f957bbb9: Status 404 returned error can't find the container with id 888e2be0941810f2e82ec53e590245e6d12c31de9871b831c0bed8a0f957bbb9
Oct 30 21:53:10.814940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.814874    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-envoy" pod "6733d22f-3aa0-4a87-949f-14bed46f42ff" (UID: "6733d22f-3aa0-4a87-949f-14bed46f42ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:10.817169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.817144    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-kube-api-access-k2x9z" pod "6733d22f-3aa0-4a87-949f-14bed46f42ff" (UID: "6733d22f-3aa0-4a87-949f-14bed46f42ff"). InnerVolumeSpecName "kube-api-access-k2x9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:10.817632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.817612    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-token" pod "6733d22f-3aa0-4a87-949f-14bed46f42ff" (UID: "6733d22f-3aa0-4a87-949f-14bed46f42ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:10.886887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.886858    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.886887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.886888    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2x9z\" (UniqueName: \"kubernetes.io/projected/6733d22f-3aa0-4a87-949f-14bed46f42ff-kube-api-access-k2x9z\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.887093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.886905    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.887093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.886918    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.887093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.886930    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6733d22f-3aa0-4a87-949f-14bed46f42ff-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.887093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.886942    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:10.887093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:10.886954    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6733d22f-3aa0-4a87-949f-14bed46f42ff-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:11.154120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:11.154085    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk" event={"ID":"92239e35-8394-4120-b2d2-1344583a8a4a","Type":"ContainerStarted","Data":"888e2be0941810f2e82ec53e590245e6d12c31de9871b831c0bed8a0f957bbb9"}
Oct 30 21:53:11.155773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:11.155749    4918 generic.go:356] "Generic (PLEG): container finished" podID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerID="e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1" exitCode=0
Oct 30 21:53:11.155878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:11.155797    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"
Oct 30 21:53:11.155878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:11.155803    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" event={"ID":"6733d22f-3aa0-4a87-949f-14bed46f42ff","Type":"ContainerDied","Data":"e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1"}
Oct 30 21:53:11.155878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:11.155838    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54" event={"ID":"6733d22f-3aa0-4a87-949f-14bed46f42ff","Type":"ContainerDied","Data":"e1371ee7eb75ae4cc77f47d8acf74ce17b83bec7b721811e7c420adf9389546e"}
Oct 30 21:53:11.155878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:11.155858    4918 scope.go:122] "RemoveContainer" containerID="e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1"
Oct 30 21:53:11.167802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:11.167780    4918 scope.go:122] "RemoveContainer" containerID="e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1"
Oct 30 21:53:11.168078 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:53:11.168055    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1\": container with ID starting with e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1 not found: ID does not exist" containerID="e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1"
Oct 30 21:53:11.168180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:11.168084    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1"} err="failed to get container status \"e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1\": rpc error: code = NotFound desc = could not find container \"e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1\": container with ID starting with e6d66af00e7de1795d6b9d95303373ce136caaffe82af6e764b90adf8c3a5cd1 not found: ID does not exist"
Oct 30 21:53:11.179202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:11.179180    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"]
Oct 30 21:53:11.190971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:11.190947    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8431de-openshift-default-57bd47ddc7-cmm54"]
Oct 30 21:53:12.168369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:12.168338    4918 generic.go:356] "Generic (PLEG): container finished" podID="92239e35-8394-4120-b2d2-1344583a8a4a" containerID="30bbbf758c12b7eeebbcc7d9c728c683dc9681c4292b703d88ecdf9d7cd0fb23" exitCode=0
Oct 30 21:53:12.168761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:12.168434    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk" event={"ID":"92239e35-8394-4120-b2d2-1344583a8a4a","Type":"ContainerDied","Data":"30bbbf758c12b7eeebbcc7d9c728c683dc9681c4292b703d88ecdf9d7cd0fb23"}
Oct 30 21:53:13.081902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:13.081864    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" path="/var/lib/kubelet/pods/6733d22f-3aa0-4a87-949f-14bed46f42ff/volumes"
Oct 30 21:53:13.359243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:13.359218    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:13.435731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:13.435673    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-xks-ca\") pod \"92239e35-8394-4120-b2d2-1344583a8a4a\" (UID: \"92239e35-8394-4120-b2d2-1344583a8a4a\") "
Oct 30 21:53:13.435948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:13.435897    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-openshift-ca\") pod \"92239e35-8394-4120-b2d2-1344583a8a4a\" (UID: \"92239e35-8394-4120-b2d2-1344583a8a4a\") "
Oct 30 21:53:13.436028 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:13.435953    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92239e35-8394-4120-b2d2-1344583a8a4a-kube-api-access-f24l6\" (UniqueName: \"kubernetes.io/projected/92239e35-8394-4120-b2d2-1344583a8a4a-kube-api-access-f24l6\") pod \"92239e35-8394-4120-b2d2-1344583a8a4a\" (UID: \"92239e35-8394-4120-b2d2-1344583a8a4a\") "
Oct 30 21:53:13.436406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:13.436364    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-openshift-ca" pod "92239e35-8394-4120-b2d2-1344583a8a4a" (UID: "92239e35-8394-4120-b2d2-1344583a8a4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:13.436908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:13.436885    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:13.440797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:13.440771    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92239e35-8394-4120-b2d2-1344583a8a4a-kube-api-access-f24l6" pod "92239e35-8394-4120-b2d2-1344583a8a4a" (UID: "92239e35-8394-4120-b2d2-1344583a8a4a"). InnerVolumeSpecName "kube-api-access-f24l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:13.440922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:13.440850    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-xks-ca" pod "92239e35-8394-4120-b2d2-1344583a8a4a" (UID: "92239e35-8394-4120-b2d2-1344583a8a4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:13.538759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:13.538722    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92239e35-8394-4120-b2d2-1344583a8a4a-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:13.538759 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:13.538755    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f24l6\" (UniqueName: \"kubernetes.io/projected/92239e35-8394-4120-b2d2-1344583a8a4a-kube-api-access-f24l6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:14.201403 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:14.201356    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"
Oct 30 21:53:14.201614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:14.201412    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk" event={"ID":"92239e35-8394-4120-b2d2-1344583a8a4a","Type":"ContainerDied","Data":"888e2be0941810f2e82ec53e590245e6d12c31de9871b831c0bed8a0f957bbb9"}
Oct 30 21:53:14.201614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:14.201493    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="888e2be0941810f2e82ec53e590245e6d12c31de9871b831c0bed8a0f957bbb9"
Oct 30 21:53:16.041835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:16.041794    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"]
Oct 30 21:53:16.052722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:16.052693    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"]
Oct 30 21:53:17.675310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.675277    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"]
Oct 30 21:53:17.676221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.676200    4918 cpu_manager.go:414] "containerMap: removing container" podUID="92239e35-8394-4120-b2d2-1344583a8a4a" containerName="revoke-keys"
Oct 30 21:53:17.676221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.676219    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92239e35-8394-4120-b2d2-1344583a8a4a" containerName="revoke-keys"
Oct 30 21:53:17.676455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.676240    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerName="istio-proxy"
Oct 30 21:53:17.676455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.676245    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerName="istio-proxy"
Oct 30 21:53:17.676455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.676417    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="92239e35-8394-4120-b2d2-1344583a8a4a" containerName="revoke-keys"
Oct 30 21:53:17.676455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.676427    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6733d22f-3aa0-4a87-949f-14bed46f42ff" containerName="istio-proxy"
Oct 30 21:53:17.680786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.680764    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:17.685870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.685851    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:53:17.687633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.687616    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:53:17.689678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.689653    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"]
Oct 30 21:53:17.777710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.777679    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c13475c-9sfnr\" (UID: \"a7dad876-777d-4252-b6ec-3cc41283b6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:17.777971 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.777948    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzs2z\" (UniqueName: \"kubernetes.io/projected/a7dad876-777d-4252-b6ec-3cc41283b6df-kube-api-access-rzs2z\") pod \"maas-api-revoke-sub-5c13475c-9sfnr\" (UID: \"a7dad876-777d-4252-b6ec-3cc41283b6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:17.778023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.777999    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c13475c-9sfnr\" (UID: \"a7dad876-777d-4252-b6ec-3cc41283b6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:17.879089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.879052    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c13475c-9sfnr\" (UID: \"a7dad876-777d-4252-b6ec-3cc41283b6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:17.879269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.879165    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c13475c-9sfnr\" (UID: \"a7dad876-777d-4252-b6ec-3cc41283b6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:17.879433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.879374    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzs2z\" (UniqueName: \"kubernetes.io/projected/a7dad876-777d-4252-b6ec-3cc41283b6df-kube-api-access-rzs2z\") pod \"maas-api-revoke-sub-5c13475c-9sfnr\" (UID: \"a7dad876-777d-4252-b6ec-3cc41283b6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:17.879918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.879893    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c13475c-9sfnr\" (UID: \"a7dad876-777d-4252-b6ec-3cc41283b6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:17.881889 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.881863    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c13475c-9sfnr\" (UID: \"a7dad876-777d-4252-b6ec-3cc41283b6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:17.896495 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:17.896471    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzs2z\" (UniqueName: \"kubernetes.io/projected/a7dad876-777d-4252-b6ec-3cc41283b6df-kube-api-access-rzs2z\") pod \"maas-api-revoke-sub-5c13475c-9sfnr\" (UID: \"a7dad876-777d-4252-b6ec-3cc41283b6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:18.000355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:18.000323    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:18.368036 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:53:18.368006    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7dad876_777d_4252_b6ec_3cc41283b6df.slice/crio-ce55a458cac131aea73b3622b111772719c8b39e0438213ae70ab6f62e6c00ae WatchSource:0}: Error finding container ce55a458cac131aea73b3622b111772719c8b39e0438213ae70ab6f62e6c00ae: Status 404 returned error can't find the container with id ce55a458cac131aea73b3622b111772719c8b39e0438213ae70ab6f62e6c00ae
Oct 30 21:53:18.368814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:18.368788    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"]
Oct 30 21:53:18.722546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:18.722519    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"]
Oct 30 21:53:18.723041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:18.722709    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" containerName="istio-proxy" containerID="cri-o://615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793" gracePeriod=30
Oct 30 21:53:19.248754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:19.248668    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr" event={"ID":"a7dad876-777d-4252-b6ec-3cc41283b6df","Type":"ContainerStarted","Data":"91e502e2fab30ae233fab47dcb5e6627d03f93291e106f437fe5201907d74678"}
Oct 30 21:53:19.248754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:19.248728    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr" event={"ID":"a7dad876-777d-4252-b6ec-3cc41283b6df","Type":"ContainerStarted","Data":"ce55a458cac131aea73b3622b111772719c8b39e0438213ae70ab6f62e6c00ae"}
Oct 30 21:53:19.269770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:19.269724    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr" podStartSLOduration=2.269709681 podStartE2EDuration="2.269709681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:19.264766894 +0000 UTC m=+3112.311326060" watchObservedRunningTime="2026-10-30 21:53:19.269709681 +0000 UTC m=+3112.316268827"
Oct 30 21:53:20.335030 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:20.334994    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"]
Oct 30 21:53:20.345898 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:20.345869    4918 generic.go:356] "Generic (PLEG): container finished" podID="a7dad876-777d-4252-b6ec-3cc41283b6df" containerID="91e502e2fab30ae233fab47dcb5e6627d03f93291e106f437fe5201907d74678" exitCode=0
Oct 30 21:53:20.346234 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:20.346043    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr" event={"ID":"a7dad876-777d-4252-b6ec-3cc41283b6df","Type":"ContainerDied","Data":"91e502e2fab30ae233fab47dcb5e6627d03f93291e106f437fe5201907d74678"}
Oct 30 21:53:21.043711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:21.043647    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg" podUID="cd30e979-c50a-4be3-9ed5-edd99eef7c67" containerName="payload-processing" containerID="cri-o://dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b" gracePeriod=25
Oct 30 21:53:21.053150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:21.053119    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95" podUID="45c3c165-44f0-4b29-ba24-01c7e16a0e22" containerName="payload-pre-processing" containerID="cri-o://6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea" gracePeriod=25
Oct 30 21:53:21.578766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:21.578740    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:21.688335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:21.688253    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-openshift-ca\") pod \"a7dad876-777d-4252-b6ec-3cc41283b6df\" (UID: \"a7dad876-777d-4252-b6ec-3cc41283b6df\") "
Oct 30 21:53:21.688539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:21.688416    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-xks-ca\") pod \"a7dad876-777d-4252-b6ec-3cc41283b6df\" (UID: \"a7dad876-777d-4252-b6ec-3cc41283b6df\") "
Oct 30 21:53:21.688539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:21.688472    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7dad876-777d-4252-b6ec-3cc41283b6df-kube-api-access-rzs2z\" (UniqueName: \"kubernetes.io/projected/a7dad876-777d-4252-b6ec-3cc41283b6df-kube-api-access-rzs2z\") pod \"a7dad876-777d-4252-b6ec-3cc41283b6df\" (UID: \"a7dad876-777d-4252-b6ec-3cc41283b6df\") "
Oct 30 21:53:21.688725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:21.688684    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-openshift-ca" pod "a7dad876-777d-4252-b6ec-3cc41283b6df" (UID: "a7dad876-777d-4252-b6ec-3cc41283b6df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:21.689099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:21.689070    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:21.690642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:21.690621    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-xks-ca" pod "a7dad876-777d-4252-b6ec-3cc41283b6df" (UID: "a7dad876-777d-4252-b6ec-3cc41283b6df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:21.690849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:21.690827    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7dad876-777d-4252-b6ec-3cc41283b6df-kube-api-access-rzs2z" pod "a7dad876-777d-4252-b6ec-3cc41283b6df" (UID: "a7dad876-777d-4252-b6ec-3cc41283b6df"). InnerVolumeSpecName "kube-api-access-rzs2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:21.790924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:21.790892    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dad876-777d-4252-b6ec-3cc41283b6df-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:21.790924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:21.790918    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzs2z\" (UniqueName: \"kubernetes.io/projected/a7dad876-777d-4252-b6ec-3cc41283b6df-kube-api-access-rzs2z\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:22.366751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:22.366724    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"
Oct 30 21:53:22.366957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:22.366759    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr" event={"ID":"a7dad876-777d-4252-b6ec-3cc41283b6df","Type":"ContainerDied","Data":"ce55a458cac131aea73b3622b111772719c8b39e0438213ae70ab6f62e6c00ae"}
Oct 30 21:53:22.366957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:22.366804    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce55a458cac131aea73b3622b111772719c8b39e0438213ae70ab6f62e6c00ae"
Oct 30 21:53:24.038433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.038365    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:53:24.134497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.134450    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-certs\") pod \"5fed745f-0165-449e-97fd-5549798ae0fc\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") "
Oct 30 21:53:24.134702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.134500    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-data\") pod \"5fed745f-0165-449e-97fd-5549798ae0fc\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") "
Oct 30 21:53:24.134702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.134601    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-kube-api-access-8d7hh\" (UniqueName: \"kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-kube-api-access-8d7hh\") pod \"5fed745f-0165-449e-97fd-5549798ae0fc\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") "
Oct 30 21:53:24.134917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.134746    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-istio-token\" (UniqueName: \"kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-istio-token\") pod \"5fed745f-0165-449e-97fd-5549798ae0fc\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") "
Oct 30 21:53:24.134917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.134785    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-certs" pod "5fed745f-0165-449e-97fd-5549798ae0fc" (UID: "5fed745f-0165-449e-97fd-5549798ae0fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:24.134917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.134847    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-socket\") pod \"5fed745f-0165-449e-97fd-5549798ae0fc\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") "
Oct 30 21:53:24.134917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.134874    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fed745f-0165-449e-97fd-5549798ae0fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fed745f-0165-449e-97fd-5549798ae0fc-istiod-ca-cert\") pod \"5fed745f-0165-449e-97fd-5549798ae0fc\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") "
Oct 30 21:53:24.135136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.134972    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fed745f-0165-449e-97fd-5549798ae0fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fed745f-0165-449e-97fd-5549798ae0fc-istio-podinfo\") pod \"5fed745f-0165-449e-97fd-5549798ae0fc\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") "
Oct 30 21:53:24.135136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.135045    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-data" pod "5fed745f-0165-449e-97fd-5549798ae0fc" (UID: "5fed745f-0165-449e-97fd-5549798ae0fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:24.135136 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.135054    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-credential-socket\") pod \"5fed745f-0165-449e-97fd-5549798ae0fc\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") "
Oct 30 21:53:24.135293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.135266    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-envoy\") pod \"5fed745f-0165-449e-97fd-5549798ae0fc\" (UID: \"5fed745f-0165-449e-97fd-5549798ae0fc\") "
Oct 30 21:53:24.135369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.135346    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-credential-socket" pod "5fed745f-0165-449e-97fd-5549798ae0fc" (UID: "5fed745f-0165-449e-97fd-5549798ae0fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:24.135948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.135823    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-socket" pod "5fed745f-0165-449e-97fd-5549798ae0fc" (UID: "5fed745f-0165-449e-97fd-5549798ae0fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:24.135948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.135918    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fed745f-0165-449e-97fd-5549798ae0fc-istiod-ca-cert" pod "5fed745f-0165-449e-97fd-5549798ae0fc" (UID: "5fed745f-0165-449e-97fd-5549798ae0fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:24.136369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.136345    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:24.136500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.136377    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fed745f-0165-449e-97fd-5549798ae0fc-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:24.136500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.136439    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:24.136500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.136458    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:24.136500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.136472    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:24.138725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.138696    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-envoy" pod "5fed745f-0165-449e-97fd-5549798ae0fc" (UID: "5fed745f-0165-449e-97fd-5549798ae0fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:24.139548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.139078    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-istio-token" pod "5fed745f-0165-449e-97fd-5549798ae0fc" (UID: "5fed745f-0165-449e-97fd-5549798ae0fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:24.139548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.139239    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-kube-api-access-8d7hh" pod "5fed745f-0165-449e-97fd-5549798ae0fc" (UID: "5fed745f-0165-449e-97fd-5549798ae0fc"). InnerVolumeSpecName "kube-api-access-8d7hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:24.139921 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.139898    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fed745f-0165-449e-97fd-5549798ae0fc-istio-podinfo" pod "5fed745f-0165-449e-97fd-5549798ae0fc" (UID: "5fed745f-0165-449e-97fd-5549798ae0fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:53:24.240414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.239568    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fed745f-0165-449e-97fd-5549798ae0fc-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:24.240414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.239605    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fed745f-0165-449e-97fd-5549798ae0fc-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:24.240414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.239620    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d7hh\" (UniqueName: \"kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-kube-api-access-8d7hh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:24.240414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.239641    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fed745f-0165-449e-97fd-5549798ae0fc-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:24.386337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.386253    4918 generic.go:356] "Generic (PLEG): container finished" podID="5fed745f-0165-449e-97fd-5549798ae0fc" containerID="615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793" exitCode=0
Oct 30 21:53:24.386337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.386308    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"
Oct 30 21:53:24.386561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.386339    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" event={"ID":"5fed745f-0165-449e-97fd-5549798ae0fc","Type":"ContainerDied","Data":"615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793"}
Oct 30 21:53:24.386561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.386437    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd" event={"ID":"5fed745f-0165-449e-97fd-5549798ae0fc","Type":"ContainerDied","Data":"2e5a93ad355b304bcf4bdb70ee9a6d0f43a48666e6779f8ad930dd87144e840b"}
Oct 30 21:53:24.386561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.386467    4918 scope.go:122] "RemoveContainer" containerID="615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793"
Oct 30 21:53:24.399431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.399403    4918 scope.go:122] "RemoveContainer" containerID="615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793"
Oct 30 21:53:24.399702 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:53:24.399679    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793\": container with ID starting with 615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793 not found: ID does not exist" containerID="615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793"
Oct 30 21:53:24.399780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.399710    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793"} err="failed to get container status \"615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793\": rpc error: code = NotFound desc = could not find container \"615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793\": container with ID starting with 615df1ce4cb7f153fd75dd78c7303e0edddb478432efeae3c7b8fb3cb0c86793 not found: ID does not exist"
Oct 30 21:53:24.416751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.416728    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"]
Oct 30 21:53:24.421508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:24.421486    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-zf2bd"]
Oct 30 21:53:25.079875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:25.079841    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" path="/var/lib/kubelet/pods/5fed745f-0165-449e-97fd-5549798ae0fc/volumes"
Oct 30 21:53:25.337428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:25.337318    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj" podUID="89194be2-23cd-4416-a72e-d0bb5f928e3f" containerName="payload-pre-processing" containerID="cri-o://ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c" gracePeriod=25
Oct 30 21:53:26.609415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:26.603683    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"]
Oct 30 21:53:26.755996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:26.749416    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d9d7b-gwrmk"]
Oct 30 21:53:27.082556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:27.082526    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92239e35-8394-4120-b2d2-1344583a8a4a" path="/var/lib/kubelet/pods/92239e35-8394-4120-b2d2-1344583a8a4a/volumes"
Oct 30 21:53:28.914883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.914849    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"]
Oct 30 21:53:28.915734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.915719    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a7dad876-777d-4252-b6ec-3cc41283b6df" containerName="revoke-keys"
Oct 30 21:53:28.915797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.915736    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7dad876-777d-4252-b6ec-3cc41283b6df" containerName="revoke-keys"
Oct 30 21:53:28.915797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.915747    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" containerName="istio-proxy"
Oct 30 21:53:28.915797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.915752    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" containerName="istio-proxy"
Oct 30 21:53:28.915937 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.915925    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7dad876-777d-4252-b6ec-3cc41283b6df" containerName="revoke-keys"
Oct 30 21:53:28.915972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.915943    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fed745f-0165-449e-97fd-5549798ae0fc" containerName="istio-proxy"
Oct 30 21:53:28.920018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.920003    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:28.925089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.924539    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:53:28.925936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.925639    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:53:28.929766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.929739    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"]
Oct 30 21:53:28.935505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.935472    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65nb9\" (UniqueName: \"kubernetes.io/projected/1bf29a31-829a-45a0-85a6-1549139db5c5-kube-api-access-65nb9\") pod \"maas-api-revoke-sub-84370ca7-6prtl\" (UID: \"1bf29a31-829a-45a0-85a6-1549139db5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:28.935623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.935548    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84370ca7-6prtl\" (UID: \"1bf29a31-829a-45a0-85a6-1549139db5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:28.935878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:28.935850    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84370ca7-6prtl\" (UID: \"1bf29a31-829a-45a0-85a6-1549139db5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:29.037896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:29.037858    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84370ca7-6prtl\" (UID: \"1bf29a31-829a-45a0-85a6-1549139db5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:29.038103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:29.038026    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65nb9\" (UniqueName: \"kubernetes.io/projected/1bf29a31-829a-45a0-85a6-1549139db5c5-kube-api-access-65nb9\") pod \"maas-api-revoke-sub-84370ca7-6prtl\" (UID: \"1bf29a31-829a-45a0-85a6-1549139db5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:29.038103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:29.038080    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84370ca7-6prtl\" (UID: \"1bf29a31-829a-45a0-85a6-1549139db5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:29.038624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:29.038599    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84370ca7-6prtl\" (UID: \"1bf29a31-829a-45a0-85a6-1549139db5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:29.040708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:29.040687    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84370ca7-6prtl\" (UID: \"1bf29a31-829a-45a0-85a6-1549139db5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:29.047130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:29.047110    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65nb9\" (UniqueName: \"kubernetes.io/projected/1bf29a31-829a-45a0-85a6-1549139db5c5-kube-api-access-65nb9\") pod \"maas-api-revoke-sub-84370ca7-6prtl\" (UID: \"1bf29a31-829a-45a0-85a6-1549139db5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:29.232425 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:53:29.228041    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:53:29.250774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:29.250013    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:29.542451 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:53:29.542421    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf29a31_829a_45a0_85a6_1549139db5c5.slice/crio-202168d8bafebc707cbc90ccbb03df336a0fadf81ff8d8d722236f86d79cb020 WatchSource:0}: Error finding container 202168d8bafebc707cbc90ccbb03df336a0fadf81ff8d8d722236f86d79cb020: Status 404 returned error can't find the container with id 202168d8bafebc707cbc90ccbb03df336a0fadf81ff8d8d722236f86d79cb020
Oct 30 21:53:29.543011 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:29.542988    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"]
Oct 30 21:53:29.565966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:29.565928    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95" podUID="45c3c165-44f0-4b29-ba24-01c7e16a0e22" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:53:29.565966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:29.565920    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg" podUID="cd30e979-c50a-4be3-9ed5-edd99eef7c67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:53:30.271241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:30.245799    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"]
Oct 30 21:53:30.297221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:30.297139    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-570f01-q6p6k"]
Oct 30 21:53:30.558064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:30.557969    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl" event={"ID":"1bf29a31-829a-45a0-85a6-1549139db5c5","Type":"ContainerStarted","Data":"ceda7095b1f022f5018b9736bc15978887963f710b84f97501b708fb8fd1fb50"}
Oct 30 21:53:30.558064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:30.558031    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl" event={"ID":"1bf29a31-829a-45a0-85a6-1549139db5c5","Type":"ContainerStarted","Data":"202168d8bafebc707cbc90ccbb03df336a0fadf81ff8d8d722236f86d79cb020"}
Oct 30 21:53:30.579939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:30.579890    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl" podStartSLOduration=2.579876264 podStartE2EDuration="2.579876264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:30.577112101 +0000 UTC m=+3123.623671268" watchObservedRunningTime="2026-10-30 21:53:30.579876264 +0000 UTC m=+3123.626435409"
Oct 30 21:53:31.081591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:31.081514    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b1ee718-1182-431f-aa3e-64275bceac18" path="/var/lib/kubelet/pods/9b1ee718-1182-431f-aa3e-64275bceac18/volumes"
Oct 30 21:53:31.568632 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:31.568594    4918 generic.go:356] "Generic (PLEG): container finished" podID="1bf29a31-829a-45a0-85a6-1549139db5c5" containerID="ceda7095b1f022f5018b9736bc15978887963f710b84f97501b708fb8fd1fb50" exitCode=0
Oct 30 21:53:31.569098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:31.568678    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl" event={"ID":"1bf29a31-829a-45a0-85a6-1549139db5c5","Type":"ContainerDied","Data":"ceda7095b1f022f5018b9736bc15978887963f710b84f97501b708fb8fd1fb50"}
Oct 30 21:53:32.751668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:32.751641    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:32.910199 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:32.910096    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-openshift-ca\") pod \"1bf29a31-829a-45a0-85a6-1549139db5c5\" (UID: \"1bf29a31-829a-45a0-85a6-1549139db5c5\") "
Oct 30 21:53:32.910622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:32.910587    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-xks-ca\") pod \"1bf29a31-829a-45a0-85a6-1549139db5c5\" (UID: \"1bf29a31-829a-45a0-85a6-1549139db5c5\") "
Oct 30 21:53:32.911203 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:32.911181    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf29a31-829a-45a0-85a6-1549139db5c5-kube-api-access-65nb9\" (UniqueName: \"kubernetes.io/projected/1bf29a31-829a-45a0-85a6-1549139db5c5-kube-api-access-65nb9\") pod \"1bf29a31-829a-45a0-85a6-1549139db5c5\" (UID: \"1bf29a31-829a-45a0-85a6-1549139db5c5\") "
Oct 30 21:53:32.915208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:32.915180    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-openshift-ca" pod "1bf29a31-829a-45a0-85a6-1549139db5c5" (UID: "1bf29a31-829a-45a0-85a6-1549139db5c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:32.918893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:32.918869    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf29a31-829a-45a0-85a6-1549139db5c5-kube-api-access-65nb9" pod "1bf29a31-829a-45a0-85a6-1549139db5c5" (UID: "1bf29a31-829a-45a0-85a6-1549139db5c5"). InnerVolumeSpecName "kube-api-access-65nb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:32.929115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:32.929083    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj" podUID="89194be2-23cd-4416-a72e-d0bb5f928e3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:53:32.941568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:32.941535    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-xks-ca" pod "1bf29a31-829a-45a0-85a6-1549139db5c5" (UID: "1bf29a31-829a-45a0-85a6-1549139db5c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:33.014312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:33.014279    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65nb9\" (UniqueName: \"kubernetes.io/projected/1bf29a31-829a-45a0-85a6-1549139db5c5-kube-api-access-65nb9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:33.014312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:33.014309    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:33.014558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:33.014320    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf29a31-829a-45a0-85a6-1549139db5c5-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:33.676476 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:33.676342    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"
Oct 30 21:53:33.677215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:33.677184    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl" event={"ID":"1bf29a31-829a-45a0-85a6-1549139db5c5","Type":"ContainerDied","Data":"202168d8bafebc707cbc90ccbb03df336a0fadf81ff8d8d722236f86d79cb020"}
Oct 30 21:53:33.677352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:33.677238    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="202168d8bafebc707cbc90ccbb03df336a0fadf81ff8d8d722236f86d79cb020"
Oct 30 21:53:34.244850 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.244818    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"]
Oct 30 21:53:34.245821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.245795    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1bf29a31-829a-45a0-85a6-1549139db5c5" containerName="revoke-keys"
Oct 30 21:53:34.245821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.245820    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf29a31-829a-45a0-85a6-1549139db5c5" containerName="revoke-keys"
Oct 30 21:53:34.246036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.245983    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf29a31-829a-45a0-85a6-1549139db5c5" containerName="revoke-keys"
Oct 30 21:53:34.250377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.250359    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:34.255691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.255609    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:53:34.256089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.256061    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:53:34.259466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.259443    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"]
Oct 30 21:53:34.291511 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.291376    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-001667-hv6fp\" (UID: \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:34.291650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.291614    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-001667-hv6fp\" (UID: \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:34.291696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.291652    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps2qj\" (UniqueName: \"kubernetes.io/projected/b69e2763-51dd-4b6e-9885-25bb8f4352ab-kube-api-access-ps2qj\") pod \"maas-api-revoke-keys-e2e-inf-b-001667-hv6fp\" (UID: \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:34.393239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.393206    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-001667-hv6fp\" (UID: \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:34.393428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.393297    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-001667-hv6fp\" (UID: \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:34.393428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.393324    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps2qj\" (UniqueName: \"kubernetes.io/projected/b69e2763-51dd-4b6e-9885-25bb8f4352ab-kube-api-access-ps2qj\") pod \"maas-api-revoke-keys-e2e-inf-b-001667-hv6fp\" (UID: \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:34.393859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.393839    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-001667-hv6fp\" (UID: \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:34.395896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.395877    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-001667-hv6fp\" (UID: \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:34.403285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.403266    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps2qj\" (UniqueName: \"kubernetes.io/projected/b69e2763-51dd-4b6e-9885-25bb8f4352ab-kube-api-access-ps2qj\") pod \"maas-api-revoke-keys-e2e-inf-b-001667-hv6fp\" (UID: \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:34.567888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.567800    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:34.848637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.848563    4918 scope.go:122] "RemoveContainer" containerID="b9f0685d6563057005a71319d1fafed00a9aa20198e94c52e3c847f04846f14a"
Oct 30 21:53:34.858405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.858362    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"]
Oct 30 21:53:34.861337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.861314    4918 scope.go:122] "RemoveContainer" containerID="107ebd331d6be44a3480c1ab37b276b1e96c8bd7a96ce8a09291d593e319c854"
Oct 30 21:53:34.863631 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:53:34.863602    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69e2763_51dd_4b6e_9885_25bb8f4352ab.slice/crio-5d5de61fc30d3cd752827c0d8d304aa08dde5fe20659bf49f6c57b257a760772 WatchSource:0}: Error finding container 5d5de61fc30d3cd752827c0d8d304aa08dde5fe20659bf49f6c57b257a760772: Status 404 returned error can't find the container with id 5d5de61fc30d3cd752827c0d8d304aa08dde5fe20659bf49f6c57b257a760772
Oct 30 21:53:34.892286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:34.892258    4918 scope.go:122] "RemoveContainer" containerID="016de08f7e249e77fc681b6d5ad8b845786522e51e923a75e786c45fbf0ab977"
Oct 30 21:53:35.759569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:35.759533    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp" event={"ID":"b69e2763-51dd-4b6e-9885-25bb8f4352ab","Type":"ContainerStarted","Data":"45978fb9c86dd1522e32d1795e96e82c1538ad847f76accdf5ad7cb7507b71b6"}
Oct 30 21:53:35.759987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:35.759627    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp" event={"ID":"b69e2763-51dd-4b6e-9885-25bb8f4352ab","Type":"ContainerStarted","Data":"5d5de61fc30d3cd752827c0d8d304aa08dde5fe20659bf49f6c57b257a760772"}
Oct 30 21:53:35.783304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:35.783244    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp" podStartSLOduration=1.783231177 podStartE2EDuration="1.783231177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:35.777587384 +0000 UTC m=+3128.824146549" watchObservedRunningTime="2026-10-30 21:53:35.783231177 +0000 UTC m=+3128.829790365"
Oct 30 21:53:36.769659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:36.769618    4918 generic.go:356] "Generic (PLEG): container finished" podID="b69e2763-51dd-4b6e-9885-25bb8f4352ab" containerID="45978fb9c86dd1522e32d1795e96e82c1538ad847f76accdf5ad7cb7507b71b6" exitCode=0
Oct 30 21:53:36.770157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:36.769731    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp" event={"ID":"b69e2763-51dd-4b6e-9885-25bb8f4352ab","Type":"ContainerDied","Data":"45978fb9c86dd1522e32d1795e96e82c1538ad847f76accdf5ad7cb7507b71b6"}
Oct 30 21:53:38.012247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.012212    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"]
Oct 30 21:53:38.022833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.022805    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:38.041848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.041286    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"]
Oct 30 21:53:38.145499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.139552    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f\" (UID: \"064dab87-e70c-4b92-9839-cad947e66df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:38.145499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.139878    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f\" (UID: \"064dab87-e70c-4b92-9839-cad947e66df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:38.145499 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.139915    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tflsb\" (UniqueName: \"kubernetes.io/projected/064dab87-e70c-4b92-9839-cad947e66df9-kube-api-access-tflsb\") pod \"maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f\" (UID: \"064dab87-e70c-4b92-9839-cad947e66df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:38.241378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.241346    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f\" (UID: \"064dab87-e70c-4b92-9839-cad947e66df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:38.241726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.241706    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f\" (UID: \"064dab87-e70c-4b92-9839-cad947e66df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:38.241824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.241742    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tflsb\" (UniqueName: \"kubernetes.io/projected/064dab87-e70c-4b92-9839-cad947e66df9-kube-api-access-tflsb\") pod \"maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f\" (UID: \"064dab87-e70c-4b92-9839-cad947e66df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:38.242922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.242898    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f\" (UID: \"064dab87-e70c-4b92-9839-cad947e66df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:38.245627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.245608    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f\" (UID: \"064dab87-e70c-4b92-9839-cad947e66df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:38.250753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.250732    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tflsb\" (UniqueName: \"kubernetes.io/projected/064dab87-e70c-4b92-9839-cad947e66df9-kube-api-access-tflsb\") pod \"maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f\" (UID: \"064dab87-e70c-4b92-9839-cad947e66df9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:38.285544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.285522    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:38.371872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.371791    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:38.444706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.444672    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-openshift-ca\") pod \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\" (UID: \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\") "
Oct 30 21:53:38.444854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.444836    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-xks-ca\") pod \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\" (UID: \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\") "
Oct 30 21:53:38.444998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.444981    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69e2763-51dd-4b6e-9885-25bb8f4352ab-kube-api-access-ps2qj\" (UniqueName: \"kubernetes.io/projected/b69e2763-51dd-4b6e-9885-25bb8f4352ab-kube-api-access-ps2qj\") pod \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\" (UID: \"b69e2763-51dd-4b6e-9885-25bb8f4352ab\") "
Oct 30 21:53:38.446129 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.445902    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-openshift-ca" pod "b69e2763-51dd-4b6e-9885-25bb8f4352ab" (UID: "b69e2763-51dd-4b6e-9885-25bb8f4352ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:38.448765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.448726    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-xks-ca" pod "b69e2763-51dd-4b6e-9885-25bb8f4352ab" (UID: "b69e2763-51dd-4b6e-9885-25bb8f4352ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:38.455459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.455432    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69e2763-51dd-4b6e-9885-25bb8f4352ab-kube-api-access-ps2qj" pod "b69e2763-51dd-4b6e-9885-25bb8f4352ab" (UID: "b69e2763-51dd-4b6e-9885-25bb8f4352ab"). InnerVolumeSpecName "kube-api-access-ps2qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:38.548227 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.548194    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:38.548402 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.548233    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69e2763-51dd-4b6e-9885-25bb8f4352ab-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:38.548402 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.548253    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps2qj\" (UniqueName: \"kubernetes.io/projected/b69e2763-51dd-4b6e-9885-25bb8f4352ab-kube-api-access-ps2qj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:38.657442 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:53:38.657406    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod064dab87_e70c_4b92_9839_cad947e66df9.slice/crio-6da81fb3c6bd897a003a23783805b6ea72f17c676c1cf6241efffc7199e0cd43 WatchSource:0}: Error finding container 6da81fb3c6bd897a003a23783805b6ea72f17c676c1cf6241efffc7199e0cd43: Status 404 returned error can't find the container with id 6da81fb3c6bd897a003a23783805b6ea72f17c676c1cf6241efffc7199e0cd43
Oct 30 21:53:38.658458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.658433    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"]
Oct 30 21:53:38.826475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.826437    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f" event={"ID":"064dab87-e70c-4b92-9839-cad947e66df9","Type":"ContainerStarted","Data":"6da81fb3c6bd897a003a23783805b6ea72f17c676c1cf6241efffc7199e0cd43"}
Oct 30 21:53:38.828008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.827988    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"
Oct 30 21:53:38.828137 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.828009    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp" event={"ID":"b69e2763-51dd-4b6e-9885-25bb8f4352ab","Type":"ContainerDied","Data":"5d5de61fc30d3cd752827c0d8d304aa08dde5fe20659bf49f6c57b257a760772"}
Oct 30 21:53:38.828137 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:38.828050    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d5de61fc30d3cd752827c0d8d304aa08dde5fe20659bf49f6c57b257a760772"
Oct 30 21:53:39.565518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.565479    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg" podUID="cd30e979-c50a-4be3-9ed5-edd99eef7c67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:53:39.566044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.566016    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95" podUID="45c3c165-44f0-4b29-ba24-01c7e16a0e22" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:53:39.839666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.839572    4918 generic.go:356] "Generic (PLEG): container finished" podID="064dab87-e70c-4b92-9839-cad947e66df9" containerID="9504bf90b9d010b1be6075ec1d7e9e7357f08f246a1647eea067b96ec7cca21f" exitCode=0
Oct 30 21:53:39.839821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.839654    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f" event={"ID":"064dab87-e70c-4b92-9839-cad947e66df9","Type":"ContainerDied","Data":"9504bf90b9d010b1be6075ec1d7e9e7357f08f246a1647eea067b96ec7cca21f"}
Oct 30 21:53:39.873906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.873877    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"]
Oct 30 21:53:39.879951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.875019    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b69e2763-51dd-4b6e-9885-25bb8f4352ab" containerName="revoke-keys"
Oct 30 21:53:39.879951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.875043    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69e2763-51dd-4b6e-9885-25bb8f4352ab" containerName="revoke-keys"
Oct 30 21:53:39.879951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.875247    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69e2763-51dd-4b6e-9885-25bb8f4352ab" containerName="revoke-keys"
Oct 30 21:53:39.888969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.884826    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:39.894456 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.894420    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"]
Oct 30 21:53:39.914433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.914358    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w\" (UID: \"5e90e932-65ff-4056-be86-6e399cb429be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:39.914591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.914531    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w\" (UID: \"5e90e932-65ff-4056-be86-6e399cb429be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:39.914724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:39.914707    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn67t\" (UniqueName: \"kubernetes.io/projected/5e90e932-65ff-4056-be86-6e399cb429be-kube-api-access-mn67t\") pod \"maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w\" (UID: \"5e90e932-65ff-4056-be86-6e399cb429be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:40.015729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:40.015689    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w\" (UID: \"5e90e932-65ff-4056-be86-6e399cb429be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:40.015905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:40.015857    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn67t\" (UniqueName: \"kubernetes.io/projected/5e90e932-65ff-4056-be86-6e399cb429be-kube-api-access-mn67t\") pod \"maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w\" (UID: \"5e90e932-65ff-4056-be86-6e399cb429be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:40.016038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:40.016025    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w\" (UID: \"5e90e932-65ff-4056-be86-6e399cb429be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:40.016709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:40.016681    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w\" (UID: \"5e90e932-65ff-4056-be86-6e399cb429be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:40.018218 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:40.018200    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w\" (UID: \"5e90e932-65ff-4056-be86-6e399cb429be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:40.024487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:40.024439    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn67t\" (UniqueName: \"kubernetes.io/projected/5e90e932-65ff-4056-be86-6e399cb429be-kube-api-access-mn67t\") pod \"maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w\" (UID: \"5e90e932-65ff-4056-be86-6e399cb429be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:40.210699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:40.210614    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:40.499006 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:53:40.498973    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e90e932_65ff_4056_be86_6e399cb429be.slice/crio-f4cd6ab2487caf02bac8301a9359a6bf637d56fcd72a7df11035a3dc50425e7e WatchSource:0}: Error finding container f4cd6ab2487caf02bac8301a9359a6bf637d56fcd72a7df11035a3dc50425e7e: Status 404 returned error can't find the container with id f4cd6ab2487caf02bac8301a9359a6bf637d56fcd72a7df11035a3dc50425e7e
Oct 30 21:53:40.500141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:40.500116    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"]
Oct 30 21:53:40.850453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:40.850287    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w" event={"ID":"5e90e932-65ff-4056-be86-6e399cb429be","Type":"ContainerStarted","Data":"f4cd6ab2487caf02bac8301a9359a6bf637d56fcd72a7df11035a3dc50425e7e"}
Oct 30 21:53:41.116284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.116250    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:41.170522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.170477    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064dab87-e70c-4b92-9839-cad947e66df9-kube-api-access-tflsb\" (UniqueName: \"kubernetes.io/projected/064dab87-e70c-4b92-9839-cad947e66df9-kube-api-access-tflsb\") pod \"064dab87-e70c-4b92-9839-cad947e66df9\" (UID: \"064dab87-e70c-4b92-9839-cad947e66df9\") "
Oct 30 21:53:41.170806 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.170785    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-openshift-ca\") pod \"064dab87-e70c-4b92-9839-cad947e66df9\" (UID: \"064dab87-e70c-4b92-9839-cad947e66df9\") "
Oct 30 21:53:41.171491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.170928    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-xks-ca\") pod \"064dab87-e70c-4b92-9839-cad947e66df9\" (UID: \"064dab87-e70c-4b92-9839-cad947e66df9\") "
Oct 30 21:53:41.171491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.171261    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-openshift-ca" pod "064dab87-e70c-4b92-9839-cad947e66df9" (UID: "064dab87-e70c-4b92-9839-cad947e66df9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:41.172162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.171807    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:41.173580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.173482    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064dab87-e70c-4b92-9839-cad947e66df9-kube-api-access-tflsb" pod "064dab87-e70c-4b92-9839-cad947e66df9" (UID: "064dab87-e70c-4b92-9839-cad947e66df9"). InnerVolumeSpecName "kube-api-access-tflsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:41.174895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.174871    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-xks-ca" pod "064dab87-e70c-4b92-9839-cad947e66df9" (UID: "064dab87-e70c-4b92-9839-cad947e66df9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:41.274876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.274802    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064dab87-e70c-4b92-9839-cad947e66df9-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:41.274876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.274838    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tflsb\" (UniqueName: \"kubernetes.io/projected/064dab87-e70c-4b92-9839-cad947e66df9-kube-api-access-tflsb\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:41.380527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.380498    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"
Oct 30 21:53:41.416602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.416574    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"
Oct 30 21:53:41.477860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.477826    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd30e979-c50a-4be3-9ed5-edd99eef7c67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd30e979-c50a-4be3-9ed5-edd99eef7c67-plugins-config-volume\") pod \"cd30e979-c50a-4be3-9ed5-edd99eef7c67\" (UID: \"cd30e979-c50a-4be3-9ed5-edd99eef7c67\") "
Oct 30 21:53:41.478151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.478129    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c3c165-44f0-4b29-ba24-01c7e16a0e22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c3c165-44f0-4b29-ba24-01c7e16a0e22-plugins-config-volume\") pod \"45c3c165-44f0-4b29-ba24-01c7e16a0e22\" (UID: \"45c3c165-44f0-4b29-ba24-01c7e16a0e22\") "
Oct 30 21:53:41.478245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.478220    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd30e979-c50a-4be3-9ed5-edd99eef7c67-kube-api-access-kpkkp\" (UniqueName: \"kubernetes.io/projected/cd30e979-c50a-4be3-9ed5-edd99eef7c67-kube-api-access-kpkkp\") pod \"cd30e979-c50a-4be3-9ed5-edd99eef7c67\" (UID: \"cd30e979-c50a-4be3-9ed5-edd99eef7c67\") "
Oct 30 21:53:41.478315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.478253    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c3c165-44f0-4b29-ba24-01c7e16a0e22-kube-api-access-swxnq\" (UniqueName: \"kubernetes.io/projected/45c3c165-44f0-4b29-ba24-01c7e16a0e22-kube-api-access-swxnq\") pod \"45c3c165-44f0-4b29-ba24-01c7e16a0e22\" (UID: \"45c3c165-44f0-4b29-ba24-01c7e16a0e22\") "
Oct 30 21:53:41.478315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.478280    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd30e979-c50a-4be3-9ed5-edd99eef7c67-plugins-config-volume" pod "cd30e979-c50a-4be3-9ed5-edd99eef7c67" (UID: "cd30e979-c50a-4be3-9ed5-edd99eef7c67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:41.478613 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.478590    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c3c165-44f0-4b29-ba24-01c7e16a0e22-plugins-config-volume" pod "45c3c165-44f0-4b29-ba24-01c7e16a0e22" (UID: "45c3c165-44f0-4b29-ba24-01c7e16a0e22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:41.478940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.478919    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45c3c165-44f0-4b29-ba24-01c7e16a0e22-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:41.479025 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.478942    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd30e979-c50a-4be3-9ed5-edd99eef7c67-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:41.480807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.480788    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd30e979-c50a-4be3-9ed5-edd99eef7c67-kube-api-access-kpkkp" pod "cd30e979-c50a-4be3-9ed5-edd99eef7c67" (UID: "cd30e979-c50a-4be3-9ed5-edd99eef7c67"). InnerVolumeSpecName "kube-api-access-kpkkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:41.480893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.480814    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c3c165-44f0-4b29-ba24-01c7e16a0e22-kube-api-access-swxnq" pod "45c3c165-44f0-4b29-ba24-01c7e16a0e22" (UID: "45c3c165-44f0-4b29-ba24-01c7e16a0e22"). InnerVolumeSpecName "kube-api-access-swxnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:41.580438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.580409    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swxnq\" (UniqueName: \"kubernetes.io/projected/45c3c165-44f0-4b29-ba24-01c7e16a0e22-kube-api-access-swxnq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:41.580438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.580435    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpkkp\" (UniqueName: \"kubernetes.io/projected/cd30e979-c50a-4be3-9ed5-edd99eef7c67-kube-api-access-kpkkp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:41.861212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.861182    4918 generic.go:356] "Generic (PLEG): container finished" podID="45c3c165-44f0-4b29-ba24-01c7e16a0e22" containerID="6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea" exitCode=0
Oct 30 21:53:41.861726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.861220    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"
Oct 30 21:53:41.861726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.861248    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95" event={"ID":"45c3c165-44f0-4b29-ba24-01c7e16a0e22","Type":"ContainerDied","Data":"6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea"}
Oct 30 21:53:41.861726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.861298    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95" event={"ID":"45c3c165-44f0-4b29-ba24-01c7e16a0e22","Type":"ContainerDied","Data":"2c1c86d7482cc5d874e589cbd7d51124d0c46b818d1c30a187bbfc2c98ef4706"}
Oct 30 21:53:41.861726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.861321    4918 scope.go:122] "RemoveContainer" containerID="6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea"
Oct 30 21:53:41.863118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.863070    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"
Oct 30 21:53:41.863118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.863088    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f" event={"ID":"064dab87-e70c-4b92-9839-cad947e66df9","Type":"ContainerDied","Data":"6da81fb3c6bd897a003a23783805b6ea72f17c676c1cf6241efffc7199e0cd43"}
Oct 30 21:53:41.863325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.863131    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6da81fb3c6bd897a003a23783805b6ea72f17c676c1cf6241efffc7199e0cd43"
Oct 30 21:53:41.871333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.871247    4918 generic.go:356] "Generic (PLEG): container finished" podID="5e90e932-65ff-4056-be86-6e399cb429be" containerID="846635e4a5dc543b69a5258bf818084994c44a9bb1a2080b26f098835175ff32" exitCode=0
Oct 30 21:53:41.871333 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.871276    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w" event={"ID":"5e90e932-65ff-4056-be86-6e399cb429be","Type":"ContainerDied","Data":"846635e4a5dc543b69a5258bf818084994c44a9bb1a2080b26f098835175ff32"}
Oct 30 21:53:41.873358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.873200    4918 generic.go:356] "Generic (PLEG): container finished" podID="cd30e979-c50a-4be3-9ed5-edd99eef7c67" containerID="dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b" exitCode=0
Oct 30 21:53:41.873358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.873250    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"
Oct 30 21:53:41.873358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.873272    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg" event={"ID":"cd30e979-c50a-4be3-9ed5-edd99eef7c67","Type":"ContainerDied","Data":"dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b"}
Oct 30 21:53:41.873358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.873309    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg" event={"ID":"cd30e979-c50a-4be3-9ed5-edd99eef7c67","Type":"ContainerDied","Data":"5817bf0cef0cd0d6268b019b7eb5ae5e087023dec73f87a40ee49e6c1a9b6e8f"}
Oct 30 21:53:41.875753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.875718    4918 scope.go:122] "RemoveContainer" containerID="6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea"
Oct 30 21:53:41.876068 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:53:41.876046    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea\": container with ID starting with 6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea not found: ID does not exist" containerID="6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea"
Oct 30 21:53:41.876147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.876088    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea"} err="failed to get container status \"6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea\": rpc error: code = NotFound desc = could not find container \"6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea\": container with ID starting with 6b94521882e9927f1617d8b36d6c3ee4c4104135e9a122ac581f473fe7f96aea not found: ID does not exist"
Oct 30 21:53:41.876147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.876109    4918 scope.go:122] "RemoveContainer" containerID="dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b"
Oct 30 21:53:41.967329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.967300    4918 scope.go:122] "RemoveContainer" containerID="dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b"
Oct 30 21:53:41.967776 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:53:41.967750    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b\": container with ID starting with dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b not found: ID does not exist" containerID="dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b"
Oct 30 21:53:41.967902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.967787    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b"} err="failed to get container status \"dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b\": rpc error: code = NotFound desc = could not find container \"dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b\": container with ID starting with dc00116e57217ea39f5fa80ffd3e2cd183cb0257b42772a3e2b45f99579bfb7b not found: ID does not exist"
Oct 30 21:53:41.995567 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:41.994739    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"]
Oct 30 21:53:42.003804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:42.003774    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-570f01-585cf46fbc-vjc95"]
Oct 30 21:53:42.008791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:42.008765    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"]
Oct 30 21:53:42.013320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:42.013292    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-570f01-bf6cccb49-f67pg"]
Oct 30 21:53:42.927186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:42.927152    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj" podUID="89194be2-23cd-4416-a72e-d0bb5f928e3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:53:43.083060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.083032    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45c3c165-44f0-4b29-ba24-01c7e16a0e22" path="/var/lib/kubelet/pods/45c3c165-44f0-4b29-ba24-01c7e16a0e22/volumes"
Oct 30 21:53:43.083646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.083624    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd30e979-c50a-4be3-9ed5-edd99eef7c67" path="/var/lib/kubelet/pods/cd30e979-c50a-4be3-9ed5-edd99eef7c67/volumes"
Oct 30 21:53:43.086200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.086181    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:43.211368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.211289    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e90e932-65ff-4056-be86-6e399cb429be-kube-api-access-mn67t\" (UniqueName: \"kubernetes.io/projected/5e90e932-65ff-4056-be86-6e399cb429be-kube-api-access-mn67t\") pod \"5e90e932-65ff-4056-be86-6e399cb429be\" (UID: \"5e90e932-65ff-4056-be86-6e399cb429be\") "
Oct 30 21:53:43.211727 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.211704    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-xks-ca\") pod \"5e90e932-65ff-4056-be86-6e399cb429be\" (UID: \"5e90e932-65ff-4056-be86-6e399cb429be\") "
Oct 30 21:53:43.211930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.211795    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-openshift-ca\") pod \"5e90e932-65ff-4056-be86-6e399cb429be\" (UID: \"5e90e932-65ff-4056-be86-6e399cb429be\") "
Oct 30 21:53:43.212209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.212180    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-openshift-ca" pod "5e90e932-65ff-4056-be86-6e399cb429be" (UID: "5e90e932-65ff-4056-be86-6e399cb429be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:43.212496 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.212479    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.213726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.213704    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e90e932-65ff-4056-be86-6e399cb429be-kube-api-access-mn67t" pod "5e90e932-65ff-4056-be86-6e399cb429be" (UID: "5e90e932-65ff-4056-be86-6e399cb429be"). InnerVolumeSpecName "kube-api-access-mn67t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:43.214046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.214031    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-xks-ca" pod "5e90e932-65ff-4056-be86-6e399cb429be" (UID: "5e90e932-65ff-4056-be86-6e399cb429be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:43.314251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.314219    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e90e932-65ff-4056-be86-6e399cb429be-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.314251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.314252    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn67t\" (UniqueName: \"kubernetes.io/projected/5e90e932-65ff-4056-be86-6e399cb429be-kube-api-access-mn67t\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:43.895047 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.895023    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"
Oct 30 21:53:43.895254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.895056    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w" event={"ID":"5e90e932-65ff-4056-be86-6e399cb429be","Type":"ContainerDied","Data":"f4cd6ab2487caf02bac8301a9359a6bf637d56fcd72a7df11035a3dc50425e7e"}
Oct 30 21:53:43.895254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:43.895099    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4cd6ab2487caf02bac8301a9359a6bf637d56fcd72a7df11035a3dc50425e7e"
Oct 30 21:53:44.398427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:44.398367    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"]
Oct 30 21:53:44.419491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:44.419131    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"]
Oct 30 21:53:45.649247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:45.649225    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"
Oct 30 21:53:45.684427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:45.680351    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89194be2-23cd-4416-a72e-d0bb5f928e3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89194be2-23cd-4416-a72e-d0bb5f928e3f-plugins-config-volume\") pod \"89194be2-23cd-4416-a72e-d0bb5f928e3f\" (UID: \"89194be2-23cd-4416-a72e-d0bb5f928e3f\") "
Oct 30 21:53:45.684427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:45.680618    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89194be2-23cd-4416-a72e-d0bb5f928e3f-kube-api-access-w22cq\" (UniqueName: \"kubernetes.io/projected/89194be2-23cd-4416-a72e-d0bb5f928e3f-kube-api-access-w22cq\") pod \"89194be2-23cd-4416-a72e-d0bb5f928e3f\" (UID: \"89194be2-23cd-4416-a72e-d0bb5f928e3f\") "
Oct 30 21:53:45.684427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:45.681922    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89194be2-23cd-4416-a72e-d0bb5f928e3f-plugins-config-volume" pod "89194be2-23cd-4416-a72e-d0bb5f928e3f" (UID: "89194be2-23cd-4416-a72e-d0bb5f928e3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:45.684427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:45.683633    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89194be2-23cd-4416-a72e-d0bb5f928e3f-kube-api-access-w22cq" pod "89194be2-23cd-4416-a72e-d0bb5f928e3f" (UID: "89194be2-23cd-4416-a72e-d0bb5f928e3f"). InnerVolumeSpecName "kube-api-access-w22cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:45.783316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:45.783281    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89194be2-23cd-4416-a72e-d0bb5f928e3f-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:45.783316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:45.783307    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w22cq\" (UniqueName: \"kubernetes.io/projected/89194be2-23cd-4416-a72e-d0bb5f928e3f-kube-api-access-w22cq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:46.011346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:46.011254    4918 generic.go:356] "Generic (PLEG): container finished" podID="89194be2-23cd-4416-a72e-d0bb5f928e3f" containerID="ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c" exitCode=0
Oct 30 21:53:46.011346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:46.011332    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj" event={"ID":"89194be2-23cd-4416-a72e-d0bb5f928e3f","Type":"ContainerDied","Data":"ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c"}
Oct 30 21:53:46.012189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:46.011438    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj" event={"ID":"89194be2-23cd-4416-a72e-d0bb5f928e3f","Type":"ContainerDied","Data":"fbb4c46145affdceb23accafc0a84848c6f29dc55300de62a79d9ee52d4708d6"}
Oct 30 21:53:46.012189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:46.011470    4918 scope.go:122] "RemoveContainer" containerID="ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c"
Oct 30 21:53:46.012189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:46.011634    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"
Oct 30 21:53:46.026039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:46.026021    4918 scope.go:122] "RemoveContainer" containerID="ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c"
Oct 30 21:53:46.026369 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:53:46.026350    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c\": container with ID starting with ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c not found: ID does not exist" containerID="ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c"
Oct 30 21:53:46.026478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:46.026380    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c"} err="failed to get container status \"ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c\": rpc error: code = NotFound desc = could not find container \"ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c\": container with ID starting with ff9e52bf41402605150db1fd3f804b113be6d45244e7093ed43d7abf7ab9565c not found: ID does not exist"
Oct 30 21:53:46.041379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:46.041356    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"]
Oct 30 21:53:46.054904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:46.054873    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d9d7b-68f4967495-244tj"]
Oct 30 21:53:47.079903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:47.079863    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89194be2-23cd-4416-a72e-d0bb5f928e3f" path="/var/lib/kubelet/pods/89194be2-23cd-4416-a72e-d0bb5f928e3f/volumes"
Oct 30 21:53:48.159413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:48.158834    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"]
Oct 30 21:53:48.171660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:48.171623    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"]
Oct 30 21:53:49.399498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:49.399436    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj" podUID="3568958b-8717-48bb-8dae-272d42ba8117" containerName="payload-processing" containerID="cri-o://af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d" gracePeriod=25
Oct 30 21:53:49.420992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:49.420954    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl" podUID="4e6cc2dd-df3e-4fa7-a212-a04bb37e5155" containerName="payload-pre-processing" containerID="cri-o://41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9" gracePeriod=25
Oct 30 21:53:49.721009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:49.720986    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"
Oct 30 21:53:49.789415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:49.782024    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3568958b-8717-48bb-8dae-272d42ba8117-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3568958b-8717-48bb-8dae-272d42ba8117-plugins-config-volume\") pod \"3568958b-8717-48bb-8dae-272d42ba8117\" (UID: \"3568958b-8717-48bb-8dae-272d42ba8117\") "
Oct 30 21:53:49.789415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:49.782107    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3568958b-8717-48bb-8dae-272d42ba8117-kube-api-access-sc9bn\" (UniqueName: \"kubernetes.io/projected/3568958b-8717-48bb-8dae-272d42ba8117-kube-api-access-sc9bn\") pod \"3568958b-8717-48bb-8dae-272d42ba8117\" (UID: \"3568958b-8717-48bb-8dae-272d42ba8117\") "
Oct 30 21:53:49.835883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:49.835825    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3568958b-8717-48bb-8dae-272d42ba8117-plugins-config-volume" pod "3568958b-8717-48bb-8dae-272d42ba8117" (UID: "3568958b-8717-48bb-8dae-272d42ba8117"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:49.859048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:49.859008    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3568958b-8717-48bb-8dae-272d42ba8117-kube-api-access-sc9bn" pod "3568958b-8717-48bb-8dae-272d42ba8117" (UID: "3568958b-8717-48bb-8dae-272d42ba8117"). InnerVolumeSpecName "kube-api-access-sc9bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:49.885317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:49.885257    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3568958b-8717-48bb-8dae-272d42ba8117-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:49.885317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:49.885295    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc9bn\" (UniqueName: \"kubernetes.io/projected/3568958b-8717-48bb-8dae-272d42ba8117-kube-api-access-sc9bn\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:50.034451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:50.034415    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"]
Oct 30 21:53:50.050633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:50.050597    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"]
Oct 30 21:53:50.052919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:50.052888    4918 generic.go:356] "Generic (PLEG): container finished" podID="3568958b-8717-48bb-8dae-272d42ba8117" containerID="af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d" exitCode=0
Oct 30 21:53:50.053057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:50.052927    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"
Oct 30 21:53:50.053057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:50.052953    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj" event={"ID":"3568958b-8717-48bb-8dae-272d42ba8117","Type":"ContainerDied","Data":"af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d"}
Oct 30 21:53:50.053057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:50.052999    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj" event={"ID":"3568958b-8717-48bb-8dae-272d42ba8117","Type":"ContainerDied","Data":"ef143b7e9f8a55e470d77d4112428f8782c62b3bbc585b44af023f59bf5cf884"}
Oct 30 21:53:50.053057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:50.053031    4918 scope.go:122] "RemoveContainer" containerID="af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d"
Oct 30 21:53:50.090412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:50.090366    4918 scope.go:122] "RemoveContainer" containerID="af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d"
Oct 30 21:53:50.114539 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:53:50.114444    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d\": container with ID starting with af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d not found: ID does not exist" containerID="af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d"
Oct 30 21:53:50.114833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:50.114798    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d"} err="failed to get container status \"af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d\": rpc error: code = NotFound desc = could not find container \"af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d\": container with ID starting with af962b3b2ef97e72401660ad5adfeda32bcd93dd25528e440073c635e636b40d not found: ID does not exist"
Oct 30 21:53:50.145779 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:50.145747    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"]
Oct 30 21:53:50.154836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:50.154806    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-001667-55b68dfd47-827xj"]
Oct 30 21:53:51.030053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:51.030020    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"]
Oct 30 21:53:51.034786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:51.034762    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7bb9d6-9btcd"]
Oct 30 21:53:51.080194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:51.080161    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093b41dd-3056-45ce-a515-c7d3fb315992" path="/var/lib/kubelet/pods/093b41dd-3056-45ce-a515-c7d3fb315992/volumes"
Oct 30 21:53:51.080710 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:51.080694    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3568958b-8717-48bb-8dae-272d42ba8117" path="/var/lib/kubelet/pods/3568958b-8717-48bb-8dae-272d42ba8117/volumes"
Oct 30 21:53:51.976547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:51.976474    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"]
Oct 30 21:53:52.069418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:52.057990    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cf51d4-zr55w"]
Oct 30 21:53:53.079591 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:53.079560    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e90e932-65ff-4056-be86-6e399cb429be" path="/var/lib/kubelet/pods/5e90e932-65ff-4056-be86-6e399cb429be/volumes"
Oct 30 21:53:53.160258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:53.160219    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w" podUID="20c16915-4492-4f87-bfca-b2f8952b0cc3" containerName="payload-processing" containerID="cri-o://d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa" gracePeriod=25
Oct 30 21:53:53.171897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:53.171863    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d" podUID="aea69cff-5ca5-49b5-b038-bf4598440e71" containerName="payload-pre-processing" containerID="cri-o://098d4189b90c7ed6a98d61c01c7087813f6228cb7488661aabc37a8404c57ea8" gracePeriod=25
Oct 30 21:53:53.445252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:53.444867    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w" podUID="20c16915-4492-4f87-bfca-b2f8952b0cc3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:53:53.445252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:53.444917    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d" podUID="aea69cff-5ca5-49b5-b038-bf4598440e71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:53:53.926272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:53.926237    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"]
Oct 30 21:53:53.926802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:53.926768    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw" podUID="2a7ce593-4a73-498b-b239-67d29a348efe" containerName="maas-api" containerID="cri-o://afd3c89b6d188d438a5b5cbcb5b8642c63c261f065a6a7fccd9e7e9bcab7890f" gracePeriod=30
Oct 30 21:53:54.493412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.492349    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"]
Oct 30 21:53:54.495339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.495012    4918 generic.go:356] "Generic (PLEG): container finished" podID="2a7ce593-4a73-498b-b239-67d29a348efe" containerID="afd3c89b6d188d438a5b5cbcb5b8642c63c261f065a6a7fccd9e7e9bcab7890f" exitCode=0
Oct 30 21:53:54.495339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.495268    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw" event={"ID":"2a7ce593-4a73-498b-b239-67d29a348efe","Type":"ContainerDied","Data":"afd3c89b6d188d438a5b5cbcb5b8642c63c261f065a6a7fccd9e7e9bcab7890f"}
Oct 30 21:53:54.498789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.498763    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-001667-hv6fp"]
Oct 30 21:53:54.582512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.576773    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:53:54.606752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.599582    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-metrics-certs\") pod \"2a7ce593-4a73-498b-b239-67d29a348efe\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") "
Oct 30 21:53:54.606752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.599709    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7ce593-4a73-498b-b239-67d29a348efe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7ce593-4a73-498b-b239-67d29a348efe-openshift-service-ca\") pod \"2a7ce593-4a73-498b-b239-67d29a348efe\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") "
Oct 30 21:53:54.606752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.599774    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-maas-api-tls\") pod \"2a7ce593-4a73-498b-b239-67d29a348efe\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") "
Oct 30 21:53:54.606752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.599951    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7ce593-4a73-498b-b239-67d29a348efe-kube-api-access-gmpc2\" (UniqueName: \"kubernetes.io/projected/2a7ce593-4a73-498b-b239-67d29a348efe-kube-api-access-gmpc2\") pod \"2a7ce593-4a73-498b-b239-67d29a348efe\" (UID: \"2a7ce593-4a73-498b-b239-67d29a348efe\") "
Oct 30 21:53:54.610807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.608732    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7ce593-4a73-498b-b239-67d29a348efe-openshift-service-ca" pod "2a7ce593-4a73-498b-b239-67d29a348efe" (UID: "2a7ce593-4a73-498b-b239-67d29a348efe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:54.635447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.634917    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-maas-api-tls" pod "2a7ce593-4a73-498b-b239-67d29a348efe" (UID: "2a7ce593-4a73-498b-b239-67d29a348efe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:54.643454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.641147    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7ce593-4a73-498b-b239-67d29a348efe-kube-api-access-gmpc2" pod "2a7ce593-4a73-498b-b239-67d29a348efe" (UID: "2a7ce593-4a73-498b-b239-67d29a348efe"). InnerVolumeSpecName "kube-api-access-gmpc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:54.643645 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.643621    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-metrics-certs" pod "2a7ce593-4a73-498b-b239-67d29a348efe" (UID: "2a7ce593-4a73-498b-b239-67d29a348efe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:54.702874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.702844    4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:54.702874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.702872    4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a7ce593-4a73-498b-b239-67d29a348efe-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:54.702874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.702881    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a7ce593-4a73-498b-b239-67d29a348efe-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:54.703192 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:54.702891    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmpc2\" (UniqueName: \"kubernetes.io/projected/2a7ce593-4a73-498b-b239-67d29a348efe-kube-api-access-gmpc2\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:55.034826 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:55.034789    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g" podUID="c7167d31-70cc-4ed1-9f13-9fea12fb1a85" containerName="payload-processing" containerID="cri-o://4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a" gracePeriod=25
Oct 30 21:53:55.051302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:55.051270    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr" podUID="ff7000f0-3e79-4852-9992-b07ec806326c" containerName="payload-pre-processing" containerID="cri-o://2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55" gracePeriod=25
Oct 30 21:53:55.079803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:55.079778    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b69e2763-51dd-4b6e-9885-25bb8f4352ab" path="/var/lib/kubelet/pods/b69e2763-51dd-4b6e-9885-25bb8f4352ab/volumes"
Oct 30 21:53:55.506563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:55.506534    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"
Oct 30 21:53:55.506563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:55.506545    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw" event={"ID":"2a7ce593-4a73-498b-b239-67d29a348efe","Type":"ContainerDied","Data":"f45370f9a165b7edb889a578bc7a51b16f7b6a856a7b90ef848a41b79628b7e2"}
Oct 30 21:53:55.507020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:55.506610    4918 scope.go:122] "RemoveContainer" containerID="afd3c89b6d188d438a5b5cbcb5b8642c63c261f065a6a7fccd9e7e9bcab7890f"
Oct 30 21:53:55.530502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:55.530481    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"]
Oct 30 21:53:55.537896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:55.537872    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-001667-c9f875749-xc4mw"]
Oct 30 21:53:55.722142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:55.722108    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl" podUID="4e6cc2dd-df3e-4fa7-a212-a04bb37e5155" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:53:57.079320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:57.079289    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7ce593-4a73-498b-b239-67d29a348efe" path="/var/lib/kubelet/pods/2a7ce593-4a73-498b-b239-67d29a348efe/volumes"
Oct 30 21:53:59.161869 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:53:59.161829    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:53:59.478214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:59.478177    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"]
Oct 30 21:53:59.478507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:59.478452    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy" containerID="cri-o://6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa" gracePeriod=30
Oct 30 21:53:59.844144 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:59.844055    4918 patch_prober.go:29] interesting pod/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:53:59.847362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:53:59.847324    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:54:00.084419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.080450    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"]
Oct 30 21:54:00.152467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.119583    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5c3268-t5xnz"]
Oct 30 21:54:00.373274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.373201    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"]
Oct 30 21:54:00.378425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.378403    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-e596df-ws77f"]
Oct 30 21:54:00.438879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.438817    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"]
Oct 30 21:54:00.439637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439622    4918 cpu_manager.go:414] "containerMap: removing container" podUID="89194be2-23cd-4416-a72e-d0bb5f928e3f" containerName="payload-pre-processing"
Oct 30 21:54:00.439697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439640    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89194be2-23cd-4416-a72e-d0bb5f928e3f" containerName="payload-pre-processing"
Oct 30 21:54:00.439697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439663    4918 cpu_manager.go:414] "containerMap: removing container" podUID="3568958b-8717-48bb-8dae-272d42ba8117" containerName="payload-processing"
Oct 30 21:54:00.439697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439668    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3568958b-8717-48bb-8dae-272d42ba8117" containerName="payload-processing"
Oct 30 21:54:00.439697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439677    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2a7ce593-4a73-498b-b239-67d29a348efe" containerName="maas-api"
Oct 30 21:54:00.439697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439681    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7ce593-4a73-498b-b239-67d29a348efe" containerName="maas-api"
Oct 30 21:54:00.439697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439698    4918 cpu_manager.go:414] "containerMap: removing container" podUID="45c3c165-44f0-4b29-ba24-01c7e16a0e22" containerName="payload-pre-processing"
Oct 30 21:54:00.439894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439704    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c3c165-44f0-4b29-ba24-01c7e16a0e22" containerName="payload-pre-processing"
Oct 30 21:54:00.439894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439716    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5e90e932-65ff-4056-be86-6e399cb429be" containerName="revoke-keys"
Oct 30 21:54:00.439894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439721    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e90e932-65ff-4056-be86-6e399cb429be" containerName="revoke-keys"
Oct 30 21:54:00.439894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439747    4918 cpu_manager.go:414] "containerMap: removing container" podUID="064dab87-e70c-4b92-9839-cad947e66df9" containerName="revoke-keys"
Oct 30 21:54:00.439894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439752    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064dab87-e70c-4b92-9839-cad947e66df9" containerName="revoke-keys"
Oct 30 21:54:00.439894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439765    4918 cpu_manager.go:414] "containerMap: removing container" podUID="cd30e979-c50a-4be3-9ed5-edd99eef7c67" containerName="payload-processing"
Oct 30 21:54:00.439894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439769    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd30e979-c50a-4be3-9ed5-edd99eef7c67" containerName="payload-processing"
Oct 30 21:54:00.439894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439883    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="064dab87-e70c-4b92-9839-cad947e66df9" containerName="revoke-keys"
Oct 30 21:54:00.439894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439891    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd30e979-c50a-4be3-9ed5-edd99eef7c67" containerName="payload-processing"
Oct 30 21:54:00.439894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439897    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="89194be2-23cd-4416-a72e-d0bb5f928e3f" containerName="payload-pre-processing"
Oct 30 21:54:00.440215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439908    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3568958b-8717-48bb-8dae-272d42ba8117" containerName="payload-processing"
Oct 30 21:54:00.440215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439913    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7ce593-4a73-498b-b239-67d29a348efe" containerName="maas-api"
Oct 30 21:54:00.440215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439921    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c3c165-44f0-4b29-ba24-01c7e16a0e22" containerName="payload-pre-processing"
Oct 30 21:54:00.440215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.439929    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e90e932-65ff-4056-be86-6e399cb429be" containerName="revoke-keys"
Oct 30 21:54:00.444195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.444180    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:00.452678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.452651    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:00.454294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.454276    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:54:00.454445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.454364    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"]
Oct 30 21:54:00.503448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.494701    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbv9\" (UniqueName: \"kubernetes.io/projected/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-kube-api-access-pjbv9\") pod \"maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk\" (UID: \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:00.503448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.494775    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk\" (UID: \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:00.503448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.494818    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk\" (UID: \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:00.596240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.596209    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbv9\" (UniqueName: \"kubernetes.io/projected/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-kube-api-access-pjbv9\") pod \"maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk\" (UID: \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:00.596431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.596259    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk\" (UID: \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:00.596431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.596291    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk\" (UID: \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:00.597077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.597004    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk\" (UID: \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:00.598843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.598821    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk\" (UID: \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:00.605298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.605279    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbv9\" (UniqueName: \"kubernetes.io/projected/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-kube-api-access-pjbv9\") pod \"maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk\" (UID: \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:00.820311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:00.820279    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:01.080949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:01.080867    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="064dab87-e70c-4b92-9839-cad947e66df9" path="/var/lib/kubelet/pods/064dab87-e70c-4b92-9839-cad947e66df9/volumes"
Oct 30 21:54:01.081586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:01.081561    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a09178f-5a3a-4437-92fb-73f0e4a5ace0" path="/var/lib/kubelet/pods/7a09178f-5a3a-4437-92fb-73f0e4a5ace0/volumes"
Oct 30 21:54:01.107502 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:01.107468    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0344f9_cbfc_4dff_b987_cfa436f1c15b.slice/crio-96889bcc039ff34d25d3198020585283ad2b78961e03154727be12488cca3a9b WatchSource:0}: Error finding container 96889bcc039ff34d25d3198020585283ad2b78961e03154727be12488cca3a9b: Status 404 returned error can't find the container with id 96889bcc039ff34d25d3198020585283ad2b78961e03154727be12488cca3a9b
Oct 30 21:54:01.107987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:01.107964    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"]
Oct 30 21:54:01.572124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:01.572088    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk" event={"ID":"cb0344f9-cbfc-4dff-b987-cfa436f1c15b","Type":"ContainerStarted","Data":"5537170f99989b4c8af5e5d85e7769bd190831ca93c4c447e4b759ee285f9784"}
Oct 30 21:54:01.572514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:01.572149    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk" event={"ID":"cb0344f9-cbfc-4dff-b987-cfa436f1c15b","Type":"ContainerStarted","Data":"96889bcc039ff34d25d3198020585283ad2b78961e03154727be12488cca3a9b"}
Oct 30 21:54:01.592228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:01.592177    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk" podStartSLOduration=1.5921637720000001 podStartE2EDuration="1.592163772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:01.591098817 +0000 UTC m=+3154.637657962" watchObservedRunningTime="2026-10-30 21:54:01.592163772 +0000 UTC m=+3154.638722916"
Oct 30 21:54:02.331403 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.331357    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"]
Oct 30 21:54:02.331628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.331607    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy" containerID="cri-o://d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b" gracePeriod=30
Oct 30 21:54:02.646417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.645139    4918 generic.go:356] "Generic (PLEG): container finished" podID="cb0344f9-cbfc-4dff-b987-cfa436f1c15b" containerID="5537170f99989b4c8af5e5d85e7769bd190831ca93c4c447e4b759ee285f9784" exitCode=0
Oct 30 21:54:02.646417 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.645278    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk" event={"ID":"cb0344f9-cbfc-4dff-b987-cfa436f1c15b","Type":"ContainerDied","Data":"5537170f99989b4c8af5e5d85e7769bd190831ca93c4c447e4b759ee285f9784"}
Oct 30 21:54:02.727276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.727237    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"]
Oct 30 21:54:02.733115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.732982    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:02.738221 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.738186    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"]
Oct 30 21:54:02.878464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.878416    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9\" (UID: \"3c666255-138a-4869-a17b-9dec42fd8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:02.878639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.878544    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9\" (UID: \"3c666255-138a-4869-a17b-9dec42fd8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:02.878783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.878764    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtlws\" (UniqueName: \"kubernetes.io/projected/3c666255-138a-4869-a17b-9dec42fd8be9-kube-api-access-vtlws\") pod \"maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9\" (UID: \"3c666255-138a-4869-a17b-9dec42fd8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:02.980193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.980162    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9\" (UID: \"3c666255-138a-4869-a17b-9dec42fd8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:02.980452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.980435    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtlws\" (UniqueName: \"kubernetes.io/projected/3c666255-138a-4869-a17b-9dec42fd8be9-kube-api-access-vtlws\") pod \"maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9\" (UID: \"3c666255-138a-4869-a17b-9dec42fd8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:02.980598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.980583    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9\" (UID: \"3c666255-138a-4869-a17b-9dec42fd8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:02.981536 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.981512    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9\" (UID: \"3c666255-138a-4869-a17b-9dec42fd8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:02.982918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.982897    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9\" (UID: \"3c666255-138a-4869-a17b-9dec42fd8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:02.989293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.989269    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtlws\" (UniqueName: \"kubernetes.io/projected/3c666255-138a-4869-a17b-9dec42fd8be9-kube-api-access-vtlws\") pod \"maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9\" (UID: \"3c666255-138a-4869-a17b-9dec42fd8be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:02.993562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.993541    4918 patch_prober.go:29] interesting pod/e2e-inf-b-001667-openshift-default-c686545cc-97cd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:54:02.993676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:02.993599    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:54:03.051648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:03.051622    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:03.336642 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:03.336601    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c666255_138a_4869_a17b_9dec42fd8be9.slice/crio-7360394281f2876e4e65de1ffd57a00fb9f9dbd477402b1234fd809da19a7e1c WatchSource:0}: Error finding container 7360394281f2876e4e65de1ffd57a00fb9f9dbd477402b1234fd809da19a7e1c: Status 404 returned error can't find the container with id 7360394281f2876e4e65de1ffd57a00fb9f9dbd477402b1234fd809da19a7e1c
Oct 30 21:54:03.337625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:03.337597    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"]
Oct 30 21:54:03.430019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:03.428101    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w" podUID="20c16915-4492-4f87-bfca-b2f8952b0cc3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:03.430019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:03.428703    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d" podUID="aea69cff-5ca5-49b5-b038-bf4598440e71" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:03.761528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:03.761485    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9" event={"ID":"3c666255-138a-4869-a17b-9dec42fd8be9","Type":"ContainerStarted","Data":"7360394281f2876e4e65de1ffd57a00fb9f9dbd477402b1234fd809da19a7e1c"}
Oct 30 21:54:04.085931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.085899    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:04.232548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.231975    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-openshift-ca\") pod \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\" (UID: \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\") "
Oct 30 21:54:04.232548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.232156    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-xks-ca\") pod \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\" (UID: \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\") "
Oct 30 21:54:04.232825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.232605    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-openshift-ca" pod "cb0344f9-cbfc-4dff-b987-cfa436f1c15b" (UID: "cb0344f9-cbfc-4dff-b987-cfa436f1c15b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:04.232890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.232870    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-kube-api-access-pjbv9\" (UniqueName: \"kubernetes.io/projected/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-kube-api-access-pjbv9\") pod \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\" (UID: \"cb0344f9-cbfc-4dff-b987-cfa436f1c15b\") "
Oct 30 21:54:04.234536 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.234507    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.267788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.267742    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-xks-ca" pod "cb0344f9-cbfc-4dff-b987-cfa436f1c15b" (UID: "cb0344f9-cbfc-4dff-b987-cfa436f1c15b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:04.271131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.271095    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-kube-api-access-pjbv9" pod "cb0344f9-cbfc-4dff-b987-cfa436f1c15b" (UID: "cb0344f9-cbfc-4dff-b987-cfa436f1c15b"). InnerVolumeSpecName "kube-api-access-pjbv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:04.336573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.336533    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.336573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.336561    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjbv9\" (UniqueName: \"kubernetes.io/projected/cb0344f9-cbfc-4dff-b987-cfa436f1c15b-kube-api-access-pjbv9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.756434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.756376    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:54:04.773173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.773147    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"
Oct 30 21:54:04.773576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.773191    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk" event={"ID":"cb0344f9-cbfc-4dff-b987-cfa436f1c15b","Type":"ContainerDied","Data":"96889bcc039ff34d25d3198020585283ad2b78961e03154727be12488cca3a9b"}
Oct 30 21:54:04.773576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.773245    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96889bcc039ff34d25d3198020585283ad2b78961e03154727be12488cca3a9b"
Oct 30 21:54:04.775246 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.775219    4918 generic.go:356] "Generic (PLEG): container finished" podID="3c666255-138a-4869-a17b-9dec42fd8be9" containerID="95744de4076f74f7d61404957d9cd350645ab69adf91a8ccb67458afcb8c8646" exitCode=0
Oct 30 21:54:04.775402 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.775294    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9" event={"ID":"3c666255-138a-4869-a17b-9dec42fd8be9","Type":"ContainerDied","Data":"95744de4076f74f7d61404957d9cd350645ab69adf91a8ccb67458afcb8c8646"}
Oct 30 21:54:04.777586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.777558    4918 generic.go:356] "Generic (PLEG): container finished" podID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerID="6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa" exitCode=0
Oct 30 21:54:04.777742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.777594    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"
Oct 30 21:54:04.777742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.777604    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" event={"ID":"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c","Type":"ContainerDied","Data":"6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa"}
Oct 30 21:54:04.777742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.777648    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864" event={"ID":"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c","Type":"ContainerDied","Data":"604441b3ab027a9b88a5572e7a0e2ab2db3863012f6306bd84df2c179158e547"}
Oct 30 21:54:04.777742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.777679    4918 scope.go:122] "RemoveContainer" containerID="6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa"
Oct 30 21:54:04.792276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.792259    4918 scope.go:122] "RemoveContainer" containerID="7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2"
Oct 30 21:54:04.808150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.808105    4918 scope.go:122] "RemoveContainer" containerID="6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa"
Oct 30 21:54:04.808530 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:04.808493    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa\": container with ID starting with 6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa not found: ID does not exist" containerID="6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa"
Oct 30 21:54:04.808709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.808669    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa"} err="failed to get container status \"6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa\": rpc error: code = NotFound desc = could not find container \"6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa\": container with ID starting with 6a34b7fb706788741305bcbe488957c5efd408cebcd248a9f8c21e56905efdaa not found: ID does not exist"
Oct 30 21:54:04.808709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.808707    4918 scope.go:122] "RemoveContainer" containerID="7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2"
Oct 30 21:54:04.809053 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:04.809027    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2\": container with ID starting with 7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2 not found: ID does not exist" containerID="7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2"
Oct 30 21:54:04.809150 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.809061    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2"} err="failed to get container status \"7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2\": rpc error: code = NotFound desc = could not find container \"7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2\": container with ID starting with 7a001e5505e1dbebb814e7a0b2e826a09d2fa2a5624b5a1f298686d04d0570f2 not found: ID does not exist"
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.845755    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-envoy\") pod \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") "
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.845805    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-credential-socket\") pod \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") "
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.845919    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-data\") pod \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") "
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.845991    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-token\" (UniqueName: \"kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-token\") pod \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") "
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.846018    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-socket\") pod \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") "
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.846061    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-kube-api-access-xvcdx\" (UniqueName: \"kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-kube-api-access-xvcdx\") pod \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") "
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.846150    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-credential-socket" pod "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" (UID: "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.846190    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istiod-ca-cert\") pod \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") "
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.846227    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-podinfo\") pod \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") "
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.846228    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-data" pod "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" (UID: "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.846346    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-certs\") pod \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\" (UID: \"a5cf8324-5cf6-49d2-8e8d-2ec76605e96c\") "
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.846572    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-socket" pod "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" (UID: "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.846600    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istiod-ca-cert" pod "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" (UID: "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.846886    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-certs" pod "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" (UID: "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.847422    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr" podUID="ff7000f0-3e79-4852-9992-b07ec806326c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.847593    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.847616    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.847630    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.847644    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.847657    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.849317    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g" podUID="c7167d31-70cc-4ed1-9f13-9fea12fb1a85" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.850136    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-token" pod "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" (UID: "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.850488    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-podinfo" pod "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" (UID: "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.851020    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-kube-api-access-xvcdx" pod "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" (UID: "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c"). InnerVolumeSpecName "kube-api-access-xvcdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:04.853453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.851273    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-envoy" pod "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" (UID: "a5cf8324-5cf6-49d2-8e8d-2ec76605e96c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:04.949569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.949540    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.949569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.949565    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.949569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.949576    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvcdx\" (UniqueName: \"kubernetes.io/projected/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-kube-api-access-xvcdx\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:04.949799 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:04.949586    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:05.112522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:05.112483    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"]
Oct 30 21:54:05.116949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:05.116927    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cf51d4-openshift-default-5fbcd5645c-6b864"]
Oct 30 21:54:05.720298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:05.720265    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl" podUID="4e6cc2dd-df3e-4fa7-a212-a04bb37e5155" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:05.961285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:05.961261    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:06.067992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:06.067959    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-openshift-ca\") pod \"3c666255-138a-4869-a17b-9dec42fd8be9\" (UID: \"3c666255-138a-4869-a17b-9dec42fd8be9\") "
Oct 30 21:54:06.068178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:06.068076    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-xks-ca\") pod \"3c666255-138a-4869-a17b-9dec42fd8be9\" (UID: \"3c666255-138a-4869-a17b-9dec42fd8be9\") "
Oct 30 21:54:06.068178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:06.068117    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c666255-138a-4869-a17b-9dec42fd8be9-kube-api-access-vtlws\" (UniqueName: \"kubernetes.io/projected/3c666255-138a-4869-a17b-9dec42fd8be9-kube-api-access-vtlws\") pod \"3c666255-138a-4869-a17b-9dec42fd8be9\" (UID: \"3c666255-138a-4869-a17b-9dec42fd8be9\") "
Oct 30 21:54:06.068365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:06.068305    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-openshift-ca" pod "3c666255-138a-4869-a17b-9dec42fd8be9" (UID: "3c666255-138a-4869-a17b-9dec42fd8be9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:06.068827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:06.068807    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:06.070352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:06.070327    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-xks-ca" pod "3c666255-138a-4869-a17b-9dec42fd8be9" (UID: "3c666255-138a-4869-a17b-9dec42fd8be9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:06.070689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:06.070664    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c666255-138a-4869-a17b-9dec42fd8be9-kube-api-access-vtlws" pod "3c666255-138a-4869-a17b-9dec42fd8be9" (UID: "3c666255-138a-4869-a17b-9dec42fd8be9"). InnerVolumeSpecName "kube-api-access-vtlws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:06.186492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:06.186337    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c666255-138a-4869-a17b-9dec42fd8be9-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:06.186492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:06.186384    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtlws\" (UniqueName: \"kubernetes.io/projected/3c666255-138a-4869-a17b-9dec42fd8be9-kube-api-access-vtlws\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:06.797904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:06.797868    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"
Oct 30 21:54:06.798120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:06.797903    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9" event={"ID":"3c666255-138a-4869-a17b-9dec42fd8be9","Type":"ContainerDied","Data":"7360394281f2876e4e65de1ffd57a00fb9f9dbd477402b1234fd809da19a7e1c"}
Oct 30 21:54:06.798120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:06.797950    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7360394281f2876e4e65de1ffd57a00fb9f9dbd477402b1234fd809da19a7e1c"
Oct 30 21:54:07.080298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.080205    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" path="/var/lib/kubelet/pods/a5cf8324-5cf6-49d2-8e8d-2ec76605e96c/volumes"
Oct 30 21:54:07.716830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.716797    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:54:07.813057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.813015    4918 generic.go:356] "Generic (PLEG): container finished" podID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerID="d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b" exitCode=0
Oct 30 21:54:07.813237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.813086    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"
Oct 30 21:54:07.813237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.813095    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" event={"ID":"119bae8a-d416-4701-bfc6-eeabc9db6ef9","Type":"ContainerDied","Data":"d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b"}
Oct 30 21:54:07.813237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.813161    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8" event={"ID":"119bae8a-d416-4701-bfc6-eeabc9db6ef9","Type":"ContainerDied","Data":"c397268bad3fc2fa1c401eb35f22297ec6ed8b587f6eb25d5ced202aeaa0a701"}
Oct 30 21:54:07.813237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.813192    4918 scope.go:122] "RemoveContainer" containerID="d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b"
Oct 30 21:54:07.819554 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.819530    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-certs\") pod \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") "
Oct 30 21:54:07.819682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.819614    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-podinfo\") pod \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") "
Oct 30 21:54:07.819755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.819702    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-envoy\") pod \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") "
Oct 30 21:54:07.819818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.819771    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-socket\") pod \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") "
Oct 30 21:54:07.819873 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.819823    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-certs" pod "119bae8a-d416-4701-bfc6-eeabc9db6ef9" (UID: "119bae8a-d416-4701-bfc6-eeabc9db6ef9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:07.819924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.819883    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istiod-ca-cert\") pod \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") "
Oct 30 21:54:07.819980 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.819969    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-data\") pod \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") "
Oct 30 21:54:07.820043 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.820021    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-token\" (UniqueName: \"kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-token\") pod \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") "
Oct 30 21:54:07.820135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.820114    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-socket" pod "119bae8a-d416-4701-bfc6-eeabc9db6ef9" (UID: "119bae8a-d416-4701-bfc6-eeabc9db6ef9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:07.820230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.820211    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-credential-socket\") pod \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") "
Oct 30 21:54:07.820298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.820254    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-data" pod "119bae8a-d416-4701-bfc6-eeabc9db6ef9" (UID: "119bae8a-d416-4701-bfc6-eeabc9db6ef9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:07.820358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.820315    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-kube-api-access-jrjlh\" (UniqueName: \"kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-kube-api-access-jrjlh\") pod \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\" (UID: \"119bae8a-d416-4701-bfc6-eeabc9db6ef9\") "
Oct 30 21:54:07.820358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.820330    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istiod-ca-cert" pod "119bae8a-d416-4701-bfc6-eeabc9db6ef9" (UID: "119bae8a-d416-4701-bfc6-eeabc9db6ef9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:07.820661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.820547    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-credential-socket" pod "119bae8a-d416-4701-bfc6-eeabc9db6ef9" (UID: "119bae8a-d416-4701-bfc6-eeabc9db6ef9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:07.821512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.821486    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.821639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.821516    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.821639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.821532    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.821639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.821549    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.821639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.821564    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.834448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.823519    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-envoy" pod "119bae8a-d416-4701-bfc6-eeabc9db6ef9" (UID: "119bae8a-d416-4701-bfc6-eeabc9db6ef9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:07.834448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.823525    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-podinfo" pod "119bae8a-d416-4701-bfc6-eeabc9db6ef9" (UID: "119bae8a-d416-4701-bfc6-eeabc9db6ef9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:07.834448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.823582    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-token" pod "119bae8a-d416-4701-bfc6-eeabc9db6ef9" (UID: "119bae8a-d416-4701-bfc6-eeabc9db6ef9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:07.834448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.823611    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-kube-api-access-jrjlh" pod "119bae8a-d416-4701-bfc6-eeabc9db6ef9" (UID: "119bae8a-d416-4701-bfc6-eeabc9db6ef9"). InnerVolumeSpecName "kube-api-access-jrjlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:07.834448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.826926    4918 scope.go:122] "RemoveContainer" containerID="8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf"
Oct 30 21:54:07.860347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.860325    4918 scope.go:122] "RemoveContainer" containerID="d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b"
Oct 30 21:54:07.860693 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:07.860668    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b\": container with ID starting with d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b not found: ID does not exist" containerID="d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b"
Oct 30 21:54:07.860750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.860702    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b"} err="failed to get container status \"d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b\": rpc error: code = NotFound desc = could not find container \"d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b\": container with ID starting with d43c6ef917c30db72aa3049f0fade562c6b5f73903ea6f557a1677e42bd4ee2b not found: ID does not exist"
Oct 30 21:54:07.860750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.860723    4918 scope.go:122] "RemoveContainer" containerID="8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf"
Oct 30 21:54:07.861033 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:07.861014    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf\": container with ID starting with 8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf not found: ID does not exist" containerID="8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf"
Oct 30 21:54:07.861101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.861043    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf"} err="failed to get container status \"8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf\": rpc error: code = NotFound desc = could not find container \"8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf\": container with ID starting with 8d0d4f7f6da4df9270ea612e1beb37eba35c93a5692192bbcf32c97b3ec916cf not found: ID does not exist"
Oct 30 21:54:07.923594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.923552    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrjlh\" (UniqueName: \"kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-kube-api-access-jrjlh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.923594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.923586    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.923594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.923602    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.923904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.923617    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/119bae8a-d416-4701-bfc6-eeabc9db6ef9-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.949987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.949954    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"]
Oct 30 21:54:07.950249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:07.950205    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy" containerID="cri-o://98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc" gracePeriod=30
Oct 30 21:54:08.299573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.299429    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"]
Oct 30 21:54:08.310571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.310516    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-001667-openshift-default-c686545cc-97cd8"]
Oct 30 21:54:08.320522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.320106    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"]
Oct 30 21:54:08.321039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.320764    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" podUID="389006f4-6c3d-4519-8744-46882675e05b" containerName="istio-proxy" containerID="cri-o://992bd4ec2e39a493c4b8d71b634628a72b5c94289ecf2771a9630fe3677dc07f" gracePeriod=30
Oct 30 21:54:08.490176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.490102    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"]
Oct 30 21:54:08.490938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.490923    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy"
Oct 30 21:54:08.490990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.490940    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy"
Oct 30 21:54:08.490990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.490952    4918 cpu_manager.go:414] "containerMap: removing container" podUID="3c666255-138a-4869-a17b-9dec42fd8be9" containerName="revoke-keys"
Oct 30 21:54:08.490990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.490957    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c666255-138a-4869-a17b-9dec42fd8be9" containerName="revoke-keys"
Oct 30 21:54:08.490990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.490971    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy"
Oct 30 21:54:08.490990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.490977    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy"
Oct 30 21:54:08.490990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.490985    4918 cpu_manager.go:414] "containerMap: removing container" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy"
Oct 30 21:54:08.490990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.490990    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy"
Oct 30 21:54:08.491215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.491017    4918 cpu_manager.go:414] "containerMap: removing container" podUID="cb0344f9-cbfc-4dff-b987-cfa436f1c15b" containerName="revoke-keys"
Oct 30 21:54:08.491215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.491022    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0344f9-cbfc-4dff-b987-cfa436f1c15b" containerName="revoke-keys"
Oct 30 21:54:08.491215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.491131    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0344f9-cbfc-4dff-b987-cfa436f1c15b" containerName="revoke-keys"
Oct 30 21:54:08.491215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.491139    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy"
Oct 30 21:54:08.491215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.491150    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy"
Oct 30 21:54:08.491215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.491157    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c666255-138a-4869-a17b-9dec42fd8be9" containerName="revoke-keys"
Oct 30 21:54:08.491215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.491164    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5cf8324-5cf6-49d2-8e8d-2ec76605e96c" containerName="istio-proxy"
Oct 30 21:54:08.496612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.496591    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:08.514788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.514761    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"]
Oct 30 21:54:08.542167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.542134    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:08.542341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.542303    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:54:08.568323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.564772    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t\" (UID: \"879ea4a6-5d54-4546-ba61-6d6745e953ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:08.604754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.601037    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t\" (UID: \"879ea4a6-5d54-4546-ba61-6d6745e953ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:08.604754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.601112    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvdt\" (UniqueName: \"kubernetes.io/projected/879ea4a6-5d54-4546-ba61-6d6745e953ae-kube-api-access-gxvdt\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t\" (UID: \"879ea4a6-5d54-4546-ba61-6d6745e953ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:08.702895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.702862    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t\" (UID: \"879ea4a6-5d54-4546-ba61-6d6745e953ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:08.703072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.702903    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvdt\" (UniqueName: \"kubernetes.io/projected/879ea4a6-5d54-4546-ba61-6d6745e953ae-kube-api-access-gxvdt\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t\" (UID: \"879ea4a6-5d54-4546-ba61-6d6745e953ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:08.703132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.703116    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t\" (UID: \"879ea4a6-5d54-4546-ba61-6d6745e953ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:08.703820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.703796    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t\" (UID: \"879ea4a6-5d54-4546-ba61-6d6745e953ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:08.705494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.705473    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t\" (UID: \"879ea4a6-5d54-4546-ba61-6d6745e953ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:08.720291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.720268    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvdt\" (UniqueName: \"kubernetes.io/projected/879ea4a6-5d54-4546-ba61-6d6745e953ae-kube-api-access-gxvdt\") pod \"maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t\" (UID: \"879ea4a6-5d54-4546-ba61-6d6745e953ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:08.812116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:08.812036    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:09.082283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.082207    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" path="/var/lib/kubelet/pods/119bae8a-d416-4701-bfc6-eeabc9db6ef9/volumes"
Oct 30 21:54:09.098067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.098040    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"]
Oct 30 21:54:09.098199 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:09.098048    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879ea4a6_5d54_4546_ba61_6d6745e953ae.slice/crio-c796757a1afc7db0a550187341935d3b8a1d0dd42ce6116fc14676812cb06d75 WatchSource:0}: Error finding container c796757a1afc7db0a550187341935d3b8a1d0dd42ce6116fc14676812cb06d75: Status 404 returned error can't find the container with id c796757a1afc7db0a550187341935d3b8a1d0dd42ce6116fc14676812cb06d75
Oct 30 21:54:09.784694 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.784670    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"
Oct 30 21:54:09.835574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.835532    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t" event={"ID":"879ea4a6-5d54-4546-ba61-6d6745e953ae","Type":"ContainerStarted","Data":"623840906c284e26940e5cb157fd9290da67aa58d2379c50ff9e5cfb7e7bb09d"}
Oct 30 21:54:09.835775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.835618    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t" event={"ID":"879ea4a6-5d54-4546-ba61-6d6745e953ae","Type":"ContainerStarted","Data":"c796757a1afc7db0a550187341935d3b8a1d0dd42ce6116fc14676812cb06d75"}
Oct 30 21:54:09.837277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.837244    4918 generic.go:356] "Generic (PLEG): container finished" podID="4e6cc2dd-df3e-4fa7-a212-a04bb37e5155" containerID="41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9" exitCode=0
Oct 30 21:54:09.837478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.837283    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"
Oct 30 21:54:09.837478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.837332    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl" event={"ID":"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155","Type":"ContainerDied","Data":"41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9"}
Oct 30 21:54:09.837478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.837380    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl" event={"ID":"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155","Type":"ContainerDied","Data":"693f378edfd586607546efd8e09e317d40e1f721f0c272ee3283043968de72dc"}
Oct 30 21:54:09.837478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.837424    4918 scope.go:122] "RemoveContainer" containerID="41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9"
Oct 30 21:54:09.852535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.852508    4918 scope.go:122] "RemoveContainer" containerID="41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9"
Oct 30 21:54:09.852881 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:09.852855    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9\": container with ID starting with 41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9 not found: ID does not exist" containerID="41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9"
Oct 30 21:54:09.852983 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.852894    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9"} err="failed to get container status \"41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9\": rpc error: code = NotFound desc = could not find container \"41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9\": container with ID starting with 41379410211ecfa3687a73c711a862c1cd5e628cd714d941be2164e6fd8922e9 not found: ID does not exist"
Oct 30 21:54:09.853951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.853915    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t" podStartSLOduration=1.853902615 podStartE2EDuration="1.853902615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:09.85172504 +0000 UTC m=+3162.898284197" watchObservedRunningTime="2026-10-30 21:54:09.853902615 +0000 UTC m=+3162.900461759"
Oct 30 21:54:09.923353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.923325    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-plugins-config-volume\") pod \"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155\" (UID: \"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155\") "
Oct 30 21:54:09.923566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.923551    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-kube-api-access-vqrzg\" (UniqueName: \"kubernetes.io/projected/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-kube-api-access-vqrzg\") pod \"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155\" (UID: \"4e6cc2dd-df3e-4fa7-a212-a04bb37e5155\") "
Oct 30 21:54:09.923800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.923775    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-plugins-config-volume" pod "4e6cc2dd-df3e-4fa7-a212-a04bb37e5155" (UID: "4e6cc2dd-df3e-4fa7-a212-a04bb37e5155"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:09.924159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.924129    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:09.925860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:09.925838    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-kube-api-access-vqrzg" pod "4e6cc2dd-df3e-4fa7-a212-a04bb37e5155" (UID: "4e6cc2dd-df3e-4fa7-a212-a04bb37e5155"). InnerVolumeSpecName "kube-api-access-vqrzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:10.025704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:10.025679    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqrzg\" (UniqueName: \"kubernetes.io/projected/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155-kube-api-access-vqrzg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:10.168279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:10.168250    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"]
Oct 30 21:54:10.180083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:10.180052    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-001667-7c5696fcf9-jv7wl"]
Oct 30 21:54:10.973571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:10.973534    4918 generic.go:356] "Generic (PLEG): container finished" podID="879ea4a6-5d54-4546-ba61-6d6745e953ae" containerID="623840906c284e26940e5cb157fd9290da67aa58d2379c50ff9e5cfb7e7bb09d" exitCode=0
Oct 30 21:54:10.974263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:10.974237    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t" event={"ID":"879ea4a6-5d54-4546-ba61-6d6745e953ae","Type":"ContainerDied","Data":"623840906c284e26940e5cb157fd9290da67aa58d2379c50ff9e5cfb7e7bb09d"}
Oct 30 21:54:11.101410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:11.100798    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e6cc2dd-df3e-4fa7-a212-a04bb37e5155" path="/var/lib/kubelet/pods/4e6cc2dd-df3e-4fa7-a212-a04bb37e5155/volumes"
Oct 30 21:54:12.264455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:12.264431    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:12.410946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:12.410866    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-openshift-ca\") pod \"879ea4a6-5d54-4546-ba61-6d6745e953ae\" (UID: \"879ea4a6-5d54-4546-ba61-6d6745e953ae\") "
Oct 30 21:54:12.410946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:12.410943    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/879ea4a6-5d54-4546-ba61-6d6745e953ae-kube-api-access-gxvdt\" (UniqueName: \"kubernetes.io/projected/879ea4a6-5d54-4546-ba61-6d6745e953ae-kube-api-access-gxvdt\") pod \"879ea4a6-5d54-4546-ba61-6d6745e953ae\" (UID: \"879ea4a6-5d54-4546-ba61-6d6745e953ae\") "
Oct 30 21:54:12.411145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:12.411041    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-xks-ca\") pod \"879ea4a6-5d54-4546-ba61-6d6745e953ae\" (UID: \"879ea4a6-5d54-4546-ba61-6d6745e953ae\") "
Oct 30 21:54:12.411293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:12.411266    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-openshift-ca" pod "879ea4a6-5d54-4546-ba61-6d6745e953ae" (UID: "879ea4a6-5d54-4546-ba61-6d6745e953ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:12.411754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:12.411677    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:12.413259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:12.413236    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879ea4a6-5d54-4546-ba61-6d6745e953ae-kube-api-access-gxvdt" pod "879ea4a6-5d54-4546-ba61-6d6745e953ae" (UID: "879ea4a6-5d54-4546-ba61-6d6745e953ae"). InnerVolumeSpecName "kube-api-access-gxvdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:12.413357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:12.413269    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-xks-ca" pod "879ea4a6-5d54-4546-ba61-6d6745e953ae" (UID: "879ea4a6-5d54-4546-ba61-6d6745e953ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:12.513598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:12.513565    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxvdt\" (UniqueName: \"kubernetes.io/projected/879ea4a6-5d54-4546-ba61-6d6745e953ae-kube-api-access-gxvdt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:12.513598 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:12.513594    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/879ea4a6-5d54-4546-ba61-6d6745e953ae-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.028097 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.028070    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"
Oct 30 21:54:13.028250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.028095    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t" event={"ID":"879ea4a6-5d54-4546-ba61-6d6745e953ae","Type":"ContainerDied","Data":"c796757a1afc7db0a550187341935d3b8a1d0dd42ce6116fc14676812cb06d75"}
Oct 30 21:54:13.028250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.028168    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c796757a1afc7db0a550187341935d3b8a1d0dd42ce6116fc14676812cb06d75"
Oct 30 21:54:13.416606 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:13.416562    4918 logging.go:55] [core] [Channel #14888 SubChannel #14889]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.18:50052", ServerName: "10.132.1.18:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.18:50052: connect: connection refused"
Oct 30 21:54:13.420828 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:13.420794    4918 logging.go:55] [core] [Channel #14890 SubChannel #14891]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.19:50052", ServerName: "10.132.1.19:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.19:50052: connect: connection refused"
Oct 30 21:54:13.683808 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.683779    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:54:13.686454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.686434    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"
Oct 30 21:54:13.777469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.773962    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-envoy\") pod \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") "
Oct 30 21:54:13.777469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.774178    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-certs\") pod \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") "
Oct 30 21:54:13.777469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.774253    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-socket\") pod \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") "
Oct 30 21:54:13.777469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.774374    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-token\" (UniqueName: \"kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-token\") pod \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") "
Oct 30 21:54:13.777469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.774502    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-data\") pod \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") "
Oct 30 21:54:13.777469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.774556    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-credential-socket\") pod \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") "
Oct 30 21:54:13.777469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.774614    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istiod-ca-cert\") pod \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") "
Oct 30 21:54:13.777469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.774715    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-podinfo\") pod \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") "
Oct 30 21:54:13.777469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.774881    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-kube-api-access-rmfvh\" (UniqueName: \"kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-kube-api-access-rmfvh\") pod \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\" (UID: \"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3\") "
Oct 30 21:54:13.777469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.774944    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c16915-4492-4f87-bfca-b2f8952b0cc3-kube-api-access-l2r9r\" (UniqueName: \"kubernetes.io/projected/20c16915-4492-4f87-bfca-b2f8952b0cc3-kube-api-access-l2r9r\") pod \"20c16915-4492-4f87-bfca-b2f8952b0cc3\" (UID: \"20c16915-4492-4f87-bfca-b2f8952b0cc3\") "
Oct 30 21:54:13.777469 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.775018    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c16915-4492-4f87-bfca-b2f8952b0cc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c16915-4492-4f87-bfca-b2f8952b0cc3-plugins-config-volume\") pod \"20c16915-4492-4f87-bfca-b2f8952b0cc3\" (UID: \"20c16915-4492-4f87-bfca-b2f8952b0cc3\") "
Oct 30 21:54:13.798833 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.798757    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c16915-4492-4f87-bfca-b2f8952b0cc3-plugins-config-volume" pod "20c16915-4492-4f87-bfca-b2f8952b0cc3" (UID: "20c16915-4492-4f87-bfca-b2f8952b0cc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:13.799217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.799196    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-data" pod "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" (UID: "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:13.801263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.801057    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-certs" pod "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" (UID: "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:13.801263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.801118    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-credential-socket" pod "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" (UID: "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:13.801263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.801177    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-socket" pod "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" (UID: "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:13.801263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.801233    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istiod-ca-cert" pod "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" (UID: "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:13.816074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.815994    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-kube-api-access-rmfvh" pod "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" (UID: "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3"). InnerVolumeSpecName "kube-api-access-rmfvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:13.817128 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.817096    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-token" pod "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" (UID: "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:13.820006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.818091    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-envoy" pod "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" (UID: "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:13.825941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.825912    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-podinfo" pod "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" (UID: "fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:13.826054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.826028    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c16915-4492-4f87-bfca-b2f8952b0cc3-kube-api-access-l2r9r" pod "20c16915-4492-4f87-bfca-b2f8952b0cc3" (UID: "20c16915-4492-4f87-bfca-b2f8952b0cc3"). InnerVolumeSpecName "kube-api-access-l2r9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:13.877073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.877041    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.877073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.877070    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.877230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.877085    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmfvh\" (UniqueName: \"kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-kube-api-access-rmfvh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.877230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.877100    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2r9r\" (UniqueName: \"kubernetes.io/projected/20c16915-4492-4f87-bfca-b2f8952b0cc3-kube-api-access-l2r9r\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.877230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.877114    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c16915-4492-4f87-bfca-b2f8952b0cc3-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.877230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.877127    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.877230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.877140    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.877230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.877154    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.877230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.877167    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.877230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.877180    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:13.877230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:13.877194    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.044316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.044237    4918 generic.go:356] "Generic (PLEG): container finished" podID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerID="98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc" exitCode=0
Oct 30 21:54:14.044316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.044299    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"
Oct 30 21:54:14.044580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.044290    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" event={"ID":"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3","Type":"ContainerDied","Data":"98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc"}
Oct 30 21:54:14.044580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.044370    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f" event={"ID":"fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3","Type":"ContainerDied","Data":"347ae8297144a67cd22c72aea0563d514d09e3d952779c2e8bdeb49c96257bff"}
Oct 30 21:54:14.044580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.044429    4918 scope.go:122] "RemoveContainer" containerID="98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc"
Oct 30 21:54:14.046650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.046625    4918 generic.go:356] "Generic (PLEG): container finished" podID="20c16915-4492-4f87-bfca-b2f8952b0cc3" containerID="d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa" exitCode=0
Oct 30 21:54:14.046777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.046678    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"
Oct 30 21:54:14.046847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.046786    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w" event={"ID":"20c16915-4492-4f87-bfca-b2f8952b0cc3","Type":"ContainerDied","Data":"d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa"}
Oct 30 21:54:14.046847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.046828    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w" event={"ID":"20c16915-4492-4f87-bfca-b2f8952b0cc3","Type":"ContainerDied","Data":"80b98e5988788dbca21f9b7a4641dbd2ae96a5aac9837bfee3942cbaebf5d7d9"}
Oct 30 21:54:14.053420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.053375    4918 generic.go:356] "Generic (PLEG): container finished" podID="aea69cff-5ca5-49b5-b038-bf4598440e71" containerID="098d4189b90c7ed6a98d61c01c7087813f6228cb7488661aabc37a8404c57ea8" exitCode=0
Oct 30 21:54:14.053526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.053502    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d" event={"ID":"aea69cff-5ca5-49b5-b038-bf4598440e71","Type":"ContainerDied","Data":"098d4189b90c7ed6a98d61c01c7087813f6228cb7488661aabc37a8404c57ea8"}
Oct 30 21:54:14.059789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.059695    4918 generic.go:356] "Generic (PLEG): container finished" podID="389006f4-6c3d-4519-8744-46882675e05b" containerID="992bd4ec2e39a493c4b8d71b634628a72b5c94289ecf2771a9630fe3677dc07f" exitCode=0
Oct 30 21:54:14.059886 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.059781    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" event={"ID":"389006f4-6c3d-4519-8744-46882675e05b","Type":"ContainerDied","Data":"992bd4ec2e39a493c4b8d71b634628a72b5c94289ecf2771a9630fe3677dc07f"}
Oct 30 21:54:14.075677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.075653    4918 scope.go:122] "RemoveContainer" containerID="3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03"
Oct 30 21:54:14.083562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.083465    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"]
Oct 30 21:54:14.090895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.090187    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e596df-openshift-default-758b446fb5-lvg9f"]
Oct 30 21:54:14.100226 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.100203    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"]
Oct 30 21:54:14.109363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.109325    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w"]
Oct 30 21:54:14.111254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.111234    4918 scope.go:122] "RemoveContainer" containerID="98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc"
Oct 30 21:54:14.111751 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:14.111727    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc\": container with ID starting with 98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc not found: ID does not exist" containerID="98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc"
Oct 30 21:54:14.111852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.111762    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc"} err="failed to get container status \"98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc\": rpc error: code = NotFound desc = could not find container \"98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc\": container with ID starting with 98b94de85cde0e5bcb6672bec8829e9298825d2c4f0e37b7280e1b68be1c56fc not found: ID does not exist"
Oct 30 21:54:14.111852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.111782    4918 scope.go:122] "RemoveContainer" containerID="3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03"
Oct 30 21:54:14.112104 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:14.112077    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03\": container with ID starting with 3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03 not found: ID does not exist" containerID="3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03"
Oct 30 21:54:14.112210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.112111    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03"} err="failed to get container status \"3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03\": rpc error: code = NotFound desc = could not find container \"3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03\": container with ID starting with 3068588fcd8d77565799bec680dab07a3314c63c3ccf3df376d7b15365378f03 not found: ID does not exist"
Oct 30 21:54:14.112210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.112133    4918 scope.go:122] "RemoveContainer" containerID="d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa"
Oct 30 21:54:14.122167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.122145    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:54:14.126307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.126285    4918 scope.go:122] "RemoveContainer" containerID="d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa"
Oct 30 21:54:14.126673 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:14.126626    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa\": container with ID starting with d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa not found: ID does not exist" containerID="d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa"
Oct 30 21:54:14.126673 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.126663    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa"} err="failed to get container status \"d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa\": rpc error: code = NotFound desc = could not find container \"d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa\": container with ID starting with d97dc749097e287af17c3a6f927380a9ac4979a95d3997d18154d85c96da6ffa not found: ID does not exist"
Oct 30 21:54:14.145885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.145868    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:54:14.190345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.190314    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-socket\") pod \"389006f4-6c3d-4519-8744-46882675e05b\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") "
Oct 30 21:54:14.190572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.190444    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-istio-token\" (UniqueName: \"kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-istio-token\") pod \"389006f4-6c3d-4519-8744-46882675e05b\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") "
Oct 30 21:54:14.190572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.190508    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-kube-api-access-2xfzj\" (UniqueName: \"kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-kube-api-access-2xfzj\") pod \"389006f4-6c3d-4519-8744-46882675e05b\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") "
Oct 30 21:54:14.190849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.190592    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389006f4-6c3d-4519-8744-46882675e05b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389006f4-6c3d-4519-8744-46882675e05b-istiod-ca-cert\") pod \"389006f4-6c3d-4519-8744-46882675e05b\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") "
Oct 30 21:54:14.190849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.190601    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-socket" pod "389006f4-6c3d-4519-8744-46882675e05b" (UID: "389006f4-6c3d-4519-8744-46882675e05b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:14.190849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.190713    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-certs\") pod \"389006f4-6c3d-4519-8744-46882675e05b\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") "
Oct 30 21:54:14.190849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.190755    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-credential-socket\") pod \"389006f4-6c3d-4519-8744-46882675e05b\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") "
Oct 30 21:54:14.191072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.190912    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/389006f4-6c3d-4519-8744-46882675e05b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389006f4-6c3d-4519-8744-46882675e05b-istio-podinfo\") pod \"389006f4-6c3d-4519-8744-46882675e05b\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") "
Oct 30 21:54:14.191072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.190961    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea69cff-5ca5-49b5-b038-bf4598440e71-kube-api-access-96vjv\" (UniqueName: \"kubernetes.io/projected/aea69cff-5ca5-49b5-b038-bf4598440e71-kube-api-access-96vjv\") pod \"aea69cff-5ca5-49b5-b038-bf4598440e71\" (UID: \"aea69cff-5ca5-49b5-b038-bf4598440e71\") "
Oct 30 21:54:14.191072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.190993    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-certs" pod "389006f4-6c3d-4519-8744-46882675e05b" (UID: "389006f4-6c3d-4519-8744-46882675e05b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:14.191072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.191003    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-envoy\") pod \"389006f4-6c3d-4519-8744-46882675e05b\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") "
Oct 30 21:54:14.191072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.191012    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389006f4-6c3d-4519-8744-46882675e05b-istiod-ca-cert" pod "389006f4-6c3d-4519-8744-46882675e05b" (UID: "389006f4-6c3d-4519-8744-46882675e05b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:14.191325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.191091    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea69cff-5ca5-49b5-b038-bf4598440e71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea69cff-5ca5-49b5-b038-bf4598440e71-plugins-config-volume\") pod \"aea69cff-5ca5-49b5-b038-bf4598440e71\" (UID: \"aea69cff-5ca5-49b5-b038-bf4598440e71\") "
Oct 30 21:54:14.191325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.191169    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-data\") pod \"389006f4-6c3d-4519-8744-46882675e05b\" (UID: \"389006f4-6c3d-4519-8744-46882675e05b\") "
Oct 30 21:54:14.191325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.191189    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-credential-socket" pod "389006f4-6c3d-4519-8744-46882675e05b" (UID: "389006f4-6c3d-4519-8744-46882675e05b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:14.192716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.191737    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-data" pod "389006f4-6c3d-4519-8744-46882675e05b" (UID: "389006f4-6c3d-4519-8744-46882675e05b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:14.192716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.191951    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea69cff-5ca5-49b5-b038-bf4598440e71-plugins-config-volume" pod "aea69cff-5ca5-49b5-b038-bf4598440e71" (UID: "aea69cff-5ca5-49b5-b038-bf4598440e71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:14.193083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.192979    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.193083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.193005    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.193083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.193021    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea69cff-5ca5-49b5-b038-bf4598440e71-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.193083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.193037    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.193083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.193050    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.193083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.193064    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389006f4-6c3d-4519-8744-46882675e05b-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.193909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.193887    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-kube-api-access-2xfzj" pod "389006f4-6c3d-4519-8744-46882675e05b" (UID: "389006f4-6c3d-4519-8744-46882675e05b"). InnerVolumeSpecName "kube-api-access-2xfzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:14.194212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.194189    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea69cff-5ca5-49b5-b038-bf4598440e71-kube-api-access-96vjv" pod "aea69cff-5ca5-49b5-b038-bf4598440e71" (UID: "aea69cff-5ca5-49b5-b038-bf4598440e71"). InnerVolumeSpecName "kube-api-access-96vjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:14.194350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.194224    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-envoy" pod "389006f4-6c3d-4519-8744-46882675e05b" (UID: "389006f4-6c3d-4519-8744-46882675e05b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:14.194976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.194953    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/389006f4-6c3d-4519-8744-46882675e05b-istio-podinfo" pod "389006f4-6c3d-4519-8744-46882675e05b" (UID: "389006f4-6c3d-4519-8744-46882675e05b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:14.196259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.196234    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-istio-token" pod "389006f4-6c3d-4519-8744-46882675e05b" (UID: "389006f4-6c3d-4519-8744-46882675e05b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:14.295082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.295005    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389006f4-6c3d-4519-8744-46882675e05b-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.295082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.295032    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96vjv\" (UniqueName: \"kubernetes.io/projected/aea69cff-5ca5-49b5-b038-bf4598440e71-kube-api-access-96vjv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.295082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.295043    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389006f4-6c3d-4519-8744-46882675e05b-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.295082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.295053    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.295082 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.295062    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xfzj\" (UniqueName: \"kubernetes.io/projected/389006f4-6c3d-4519-8744-46882675e05b-kube-api-access-2xfzj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:14.416560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.416521    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d" podUID="aea69cff-5ca5-49b5-b038-bf4598440e71" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.18:50052\" within 1s: context deadline exceeded"
Oct 30 21:54:14.416772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.416611    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:54:14.435423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.427332    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-e596df-559989b645-xb22w" podUID="20c16915-4492-4f87-bfca-b2f8952b0cc3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.19:50052\" within 1s: context deadline exceeded"
Oct 30 21:54:14.435423 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:14.427878    4918 logging.go:55] [core] [Channel #14890 SubChannel #14891]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.19:50052", ServerName: "10.132.1.19:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.19:50052: operation was canceled"
Oct 30 21:54:14.847730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.847695    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g" podUID="c7167d31-70cc-4ed1-9f13-9fea12fb1a85" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:14.847888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:14.847798    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr" podUID="ff7000f0-3e79-4852-9992-b07ec806326c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:15.072133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.072104    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"
Oct 30 21:54:15.073866 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.073844    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"
Oct 30 21:54:15.083085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.083062    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c16915-4492-4f87-bfca-b2f8952b0cc3" path="/var/lib/kubelet/pods/20c16915-4492-4f87-bfca-b2f8952b0cc3/volumes"
Oct 30 21:54:15.083746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.083730    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" path="/var/lib/kubelet/pods/fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3/volumes"
Oct 30 21:54:15.084617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.084579    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d" event={"ID":"aea69cff-5ca5-49b5-b038-bf4598440e71","Type":"ContainerDied","Data":"7e654b3c964d74afeb37b2a4f0a18eacf542f512553939b01711b932ace8ad02"}
Oct 30 21:54:15.084735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.084632    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2" event={"ID":"389006f4-6c3d-4519-8744-46882675e05b","Type":"ContainerDied","Data":"94b39ea922aff009c83a659d64ee5681164bffac0572fdef453caa1b2a22f17b"}
Oct 30 21:54:15.084735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.084661    4918 scope.go:122] "RemoveContainer" containerID="098d4189b90c7ed6a98d61c01c7087813f6228cb7488661aabc37a8404c57ea8"
Oct 30 21:54:15.166880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.166835    4918 scope.go:122] "RemoveContainer" containerID="992bd4ec2e39a493c4b8d71b634628a72b5c94289ecf2771a9630fe3677dc07f"
Oct 30 21:54:15.185730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.185699    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"]
Oct 30 21:54:15.191130 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.190621    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e596df-6957f99c4c-k4j6d"]
Oct 30 21:54:15.279719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.279465    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"]
Oct 30 21:54:15.290324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.290288    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-468m2"]
Oct 30 21:54:15.419188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.419163    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"
Oct 30 21:54:15.445911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.445890    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"
Oct 30 21:54:15.569726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.569691    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff7000f0-3e79-4852-9992-b07ec806326c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7000f0-3e79-4852-9992-b07ec806326c-plugins-config-volume\") pod \"ff7000f0-3e79-4852-9992-b07ec806326c\" (UID: \"ff7000f0-3e79-4852-9992-b07ec806326c\") "
Oct 30 21:54:15.569909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.569786    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-plugins-config-volume\") pod \"c7167d31-70cc-4ed1-9f13-9fea12fb1a85\" (UID: \"c7167d31-70cc-4ed1-9f13-9fea12fb1a85\") "
Oct 30 21:54:15.569909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.569855    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-kube-api-access-94csr\" (UniqueName: \"kubernetes.io/projected/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-kube-api-access-94csr\") pod \"c7167d31-70cc-4ed1-9f13-9fea12fb1a85\" (UID: \"c7167d31-70cc-4ed1-9f13-9fea12fb1a85\") "
Oct 30 21:54:15.570214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.570191    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff7000f0-3e79-4852-9992-b07ec806326c-kube-api-access-4fs9n\" (UniqueName: \"kubernetes.io/projected/ff7000f0-3e79-4852-9992-b07ec806326c-kube-api-access-4fs9n\") pod \"ff7000f0-3e79-4852-9992-b07ec806326c\" (UID: \"ff7000f0-3e79-4852-9992-b07ec806326c\") "
Oct 30 21:54:15.570298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.570268    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff7000f0-3e79-4852-9992-b07ec806326c-plugins-config-volume" pod "ff7000f0-3e79-4852-9992-b07ec806326c" (UID: "ff7000f0-3e79-4852-9992-b07ec806326c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:15.570363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.570299    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-plugins-config-volume" pod "c7167d31-70cc-4ed1-9f13-9fea12fb1a85" (UID: "c7167d31-70cc-4ed1-9f13-9fea12fb1a85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:15.571040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.571014    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7000f0-3e79-4852-9992-b07ec806326c-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:15.571142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.571042    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:15.572665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.572645    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7000f0-3e79-4852-9992-b07ec806326c-kube-api-access-4fs9n" pod "ff7000f0-3e79-4852-9992-b07ec806326c" (UID: "ff7000f0-3e79-4852-9992-b07ec806326c"). InnerVolumeSpecName "kube-api-access-4fs9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:15.572746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.572695    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-kube-api-access-94csr" pod "c7167d31-70cc-4ed1-9f13-9fea12fb1a85" (UID: "c7167d31-70cc-4ed1-9f13-9fea12fb1a85"). InnerVolumeSpecName "kube-api-access-94csr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:15.672924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.672843    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fs9n\" (UniqueName: \"kubernetes.io/projected/ff7000f0-3e79-4852-9992-b07ec806326c-kube-api-access-4fs9n\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:15.672924 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:15.672872    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94csr\" (UniqueName: \"kubernetes.io/projected/c7167d31-70cc-4ed1-9f13-9fea12fb1a85-kube-api-access-94csr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:16.145679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.145640    4918 generic.go:356] "Generic (PLEG): container finished" podID="c7167d31-70cc-4ed1-9f13-9fea12fb1a85" containerID="4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a" exitCode=0
Oct 30 21:54:16.145952 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.145721    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"
Oct 30 21:54:16.146038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.145728    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g" event={"ID":"c7167d31-70cc-4ed1-9f13-9fea12fb1a85","Type":"ContainerDied","Data":"4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a"}
Oct 30 21:54:16.146104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.146090    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g" event={"ID":"c7167d31-70cc-4ed1-9f13-9fea12fb1a85","Type":"ContainerDied","Data":"fc303a371df505c4108f55ec15032a2a0ba961a309459afbd491d35e4ca36b3a"}
Oct 30 21:54:16.146148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.146123    4918 scope.go:122] "RemoveContainer" containerID="4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a"
Oct 30 21:54:16.148837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.148813    4918 generic.go:356] "Generic (PLEG): container finished" podID="ff7000f0-3e79-4852-9992-b07ec806326c" containerID="2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55" exitCode=0
Oct 30 21:54:16.149060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.148898    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"
Oct 30 21:54:16.149905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.148917    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr" event={"ID":"ff7000f0-3e79-4852-9992-b07ec806326c","Type":"ContainerDied","Data":"2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55"}
Oct 30 21:54:16.149905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.149770    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr" event={"ID":"ff7000f0-3e79-4852-9992-b07ec806326c","Type":"ContainerDied","Data":"dc64a8eb431bbb714b32e523e0b49da3ccc7e89571c55dd7eb2cd54e7ff8e7d4"}
Oct 30 21:54:16.162423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.162359    4918 scope.go:122] "RemoveContainer" containerID="4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a"
Oct 30 21:54:16.162784 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:16.162763    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a\": container with ID starting with 4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a not found: ID does not exist" containerID="4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a"
Oct 30 21:54:16.162864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.162797    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a"} err="failed to get container status \"4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a\": rpc error: code = NotFound desc = could not find container \"4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a\": container with ID starting with 4241511a8887d73eb02d0733a0b05ae1011b5c17d539a15d6012e9b966bb9b6a not found: ID does not exist"
Oct 30 21:54:16.162864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.162821    4918 scope.go:122] "RemoveContainer" containerID="2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55"
Oct 30 21:54:16.174774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.174757    4918 scope.go:122] "RemoveContainer" containerID="2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55"
Oct 30 21:54:16.175166 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:16.175127    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55\": container with ID starting with 2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55 not found: ID does not exist" containerID="2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55"
Oct 30 21:54:16.175293 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.175161    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55"} err="failed to get container status \"2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55\": rpc error: code = NotFound desc = could not find container \"2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55\": container with ID starting with 2ccda0cd2b12495788fa3e298378e2d2e8058f082cab8ead9782511c9564ec55 not found: ID does not exist"
Oct 30 21:54:16.181193 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.181167    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"]
Oct 30 21:54:16.193133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.193099    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cf51d4-675f689679-c8kcr"]
Oct 30 21:54:16.220132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.220088    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"]
Oct 30 21:54:16.225026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:16.225001    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cf51d4-f8fdbb499-cw75g"]
Oct 30 21:54:17.079785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:17.079754    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389006f4-6c3d-4519-8744-46882675e05b" path="/var/lib/kubelet/pods/389006f4-6c3d-4519-8744-46882675e05b/volumes"
Oct 30 21:54:17.080263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:17.080243    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea69cff-5ca5-49b5-b038-bf4598440e71" path="/var/lib/kubelet/pods/aea69cff-5ca5-49b5-b038-bf4598440e71/volumes"
Oct 30 21:54:17.080695 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:17.080681    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7167d31-70cc-4ed1-9f13-9fea12fb1a85" path="/var/lib/kubelet/pods/c7167d31-70cc-4ed1-9f13-9fea12fb1a85/volumes"
Oct 30 21:54:17.081419 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:17.081381    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff7000f0-3e79-4852-9992-b07ec806326c" path="/var/lib/kubelet/pods/ff7000f0-3e79-4852-9992-b07ec806326c/volumes"
Oct 30 21:54:18.569189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:18.569157    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"]
Oct 30 21:54:18.586426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:18.586378    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"]
Oct 30 21:54:19.434213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:19.434182    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"]
Oct 30 21:54:19.438672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:19.438646    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-87ae68-7qgpk"]
Oct 30 21:54:20.704999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:20.704967    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"]
Oct 30 21:54:20.709500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:20.709476    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8072a0-qdkj9"]
Oct 30 21:54:21.080288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:21.080251    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c666255-138a-4869-a17b-9dec42fd8be9" path="/var/lib/kubelet/pods/3c666255-138a-4869-a17b-9dec42fd8be9/volumes"
Oct 30 21:54:21.080790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:21.080776    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb0344f9-cbfc-4dff-b987-cfa436f1c15b" path="/var/lib/kubelet/pods/cb0344f9-cbfc-4dff-b987-cfa436f1c15b/volumes"
Oct 30 21:54:22.026509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:22.026464    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"]
Oct 30 21:54:22.030929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:22.030901    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"]
Oct 30 21:54:22.034876 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:22.034852    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c2d559a-trwcb"]
Oct 30 21:54:22.038320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:22.038298    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a4b240-htsl2"]
Oct 30 21:54:23.093764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:23.093735    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0138ea77-3423-43be-abbb-404ea9e94c8c" path="/var/lib/kubelet/pods/0138ea77-3423-43be-abbb-404ea9e94c8c/volumes"
Oct 30 21:54:23.094277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:23.094261    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69611331-493d-459d-83b1-e977506590af" path="/var/lib/kubelet/pods/69611331-493d-459d-83b1-e977506590af/volumes"
Oct 30 21:54:23.569716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:23.569675    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr" podUID="724e33b5-e637-47fb-b63f-dab247b15cd5" containerName="payload-processing" containerID="cri-o://2dea06b0a1433e40f2324ed25e7b8cf1acc515ec6b65e4603cefaa53990bdaee" gracePeriod=25
Oct 30 21:54:23.587526 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:23.587471    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc" podUID="3b022d69-5d42-42b2-bb5e-bbb98c4cedb8" containerName="payload-pre-processing" containerID="cri-o://c0537919f97d123320641e8e998c93579c889a6a95438d483ccca70f141a603c" gracePeriod=25
Oct 30 21:54:26.367823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.367792    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"]
Oct 30 21:54:26.368284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.368005    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy" containerID="cri-o://84d9f022521d2ef3b21f4d4119840f2fc525110c4b365619215477ed1c1d3be2" gracePeriod=30
Oct 30 21:54:26.834834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.834794    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"]
Oct 30 21:54:26.835643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835619    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ff7000f0-3e79-4852-9992-b07ec806326c" containerName="payload-pre-processing"
Oct 30 21:54:26.835643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835642    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff7000f0-3e79-4852-9992-b07ec806326c" containerName="payload-pre-processing"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835660    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4e6cc2dd-df3e-4fa7-a212-a04bb37e5155" containerName="payload-pre-processing"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835669    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e6cc2dd-df3e-4fa7-a212-a04bb37e5155" containerName="payload-pre-processing"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835691    4918 cpu_manager.go:414] "containerMap: removing container" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835699    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835712    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835719    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835746    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c7167d31-70cc-4ed1-9f13-9fea12fb1a85" containerName="payload-processing"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835753    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7167d31-70cc-4ed1-9f13-9fea12fb1a85" containerName="payload-processing"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835769    4918 cpu_manager.go:414] "containerMap: removing container" podUID="aea69cff-5ca5-49b5-b038-bf4598440e71" containerName="payload-pre-processing"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835775    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea69cff-5ca5-49b5-b038-bf4598440e71" containerName="payload-pre-processing"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835797    4918 cpu_manager.go:414] "containerMap: removing container" podUID="20c16915-4492-4f87-bfca-b2f8952b0cc3" containerName="payload-processing"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835803    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c16915-4492-4f87-bfca-b2f8952b0cc3" containerName="payload-processing"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835822    4918 cpu_manager.go:414] "containerMap: removing container" podUID="879ea4a6-5d54-4546-ba61-6d6745e953ae" containerName="revoke-keys"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835829    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="879ea4a6-5d54-4546-ba61-6d6745e953ae" containerName="revoke-keys"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835852    4918 cpu_manager.go:414] "containerMap: removing container" podUID="389006f4-6c3d-4519-8744-46882675e05b" containerName="istio-proxy"
Oct 30 21:54:26.835863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.835859    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389006f4-6c3d-4519-8744-46882675e05b" containerName="istio-proxy"
Oct 30 21:54:26.836765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.836038    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy"
Oct 30 21:54:26.836765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.836054    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7167d31-70cc-4ed1-9f13-9fea12fb1a85" containerName="payload-processing"
Oct 30 21:54:26.836765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.836065    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea69cff-5ca5-49b5-b038-bf4598440e71" containerName="payload-pre-processing"
Oct 30 21:54:26.836765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.836078    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c16915-4492-4f87-bfca-b2f8952b0cc3" containerName="payload-processing"
Oct 30 21:54:26.836765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.836089    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="119bae8a-d416-4701-bfc6-eeabc9db6ef9" containerName="istio-proxy"
Oct 30 21:54:26.836765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.836098    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy"
Oct 30 21:54:26.836765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.836108    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="879ea4a6-5d54-4546-ba61-6d6745e953ae" containerName="revoke-keys"
Oct 30 21:54:26.836765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.836120    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="389006f4-6c3d-4519-8744-46882675e05b" containerName="istio-proxy"
Oct 30 21:54:26.836765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.836132    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff7000f0-3e79-4852-9992-b07ec806326c" containerName="payload-pre-processing"
Oct 30 21:54:26.836765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.836141    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e6cc2dd-df3e-4fa7-a212-a04bb37e5155" containerName="payload-pre-processing"
Oct 30 21:54:26.842140 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.842120    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:26.855616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.855593    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"]
Oct 30 21:54:26.941685 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.941645    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl\" (UID: \"652dc98f-92b3-4576-9efe-c5e5c521da9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:26.941874 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.941851    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl\" (UID: \"652dc98f-92b3-4576-9efe-c5e5c521da9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:26.941945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.941885    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k7tl\" (UniqueName: \"kubernetes.io/projected/652dc98f-92b3-4576-9efe-c5e5c521da9e-kube-api-access-5k7tl\") pod \"maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl\" (UID: \"652dc98f-92b3-4576-9efe-c5e5c521da9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:26.947593 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.947572    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:26.947708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:26.947646    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:54:27.043270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:27.043237    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl\" (UID: \"652dc98f-92b3-4576-9efe-c5e5c521da9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:27.043449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:27.043275    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7tl\" (UniqueName: \"kubernetes.io/projected/652dc98f-92b3-4576-9efe-c5e5c521da9e-kube-api-access-5k7tl\") pod \"maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl\" (UID: \"652dc98f-92b3-4576-9efe-c5e5c521da9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:27.043512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:27.043492    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl\" (UID: \"652dc98f-92b3-4576-9efe-c5e5c521da9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:27.044096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:27.044070    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl\" (UID: \"652dc98f-92b3-4576-9efe-c5e5c521da9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:27.045804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:27.045786    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl\" (UID: \"652dc98f-92b3-4576-9efe-c5e5c521da9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:27.057918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:27.057892    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7tl\" (UniqueName: \"kubernetes.io/projected/652dc98f-92b3-4576-9efe-c5e5c521da9e-kube-api-access-5k7tl\") pod \"maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl\" (UID: \"652dc98f-92b3-4576-9efe-c5e5c521da9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:27.165262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:27.165187    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:54:27.174252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:27.174221    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:27.525426 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:27.525362    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652dc98f_92b3_4576_9efe_c5e5c521da9e.slice/crio-7527b0a9626893b1f26add1c642f3174bdede1186ec56120e2bf66f4e887447d WatchSource:0}: Error finding container 7527b0a9626893b1f26add1c642f3174bdede1186ec56120e2bf66f4e887447d: Status 404 returned error can't find the container with id 7527b0a9626893b1f26add1c642f3174bdede1186ec56120e2bf66f4e887447d
Oct 30 21:54:27.525716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:27.525660    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"]
Oct 30 21:54:28.455786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:28.455750    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl" event={"ID":"652dc98f-92b3-4576-9efe-c5e5c521da9e","Type":"ContainerStarted","Data":"3ef5c9235aaf9e42eee27335e8788e136ccd74101bd2443c52349214ad029407"}
Oct 30 21:54:28.455981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:28.455816    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl" event={"ID":"652dc98f-92b3-4576-9efe-c5e5c521da9e","Type":"ContainerStarted","Data":"7527b0a9626893b1f26add1c642f3174bdede1186ec56120e2bf66f4e887447d"}
Oct 30 21:54:28.487155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:28.487090    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl" podStartSLOduration=2.487069947 podStartE2EDuration="2.487069947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:28.482893585 +0000 UTC m=+3181.529452743" watchObservedRunningTime="2026-10-30 21:54:28.487069947 +0000 UTC m=+3181.533629094"
Oct 30 21:54:28.936702 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:28.936668    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"]
Oct 30 21:54:28.937500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:28.937483    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy"
Oct 30 21:54:28.937549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:28.937500    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc7885c4-7a7c-4ba2-bc4d-0ae3fda7abb3" containerName="istio-proxy"
Oct 30 21:54:28.941302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:28.941285    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:28.957003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:28.956972    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"]
Oct 30 21:54:28.995732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:28.995702    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r\" (UID: \"724e2df0-ad34-4ddc-803f-c18987837f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:28.995918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:28.995773    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh7wk\" (UniqueName: \"kubernetes.io/projected/724e2df0-ad34-4ddc-803f-c18987837f70-kube-api-access-nh7wk\") pod \"maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r\" (UID: \"724e2df0-ad34-4ddc-803f-c18987837f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:28.996056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:28.996034    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r\" (UID: \"724e2df0-ad34-4ddc-803f-c18987837f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:29.097753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:29.097720    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r\" (UID: \"724e2df0-ad34-4ddc-803f-c18987837f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:29.097974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:29.097891    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r\" (UID: \"724e2df0-ad34-4ddc-803f-c18987837f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:29.098043 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:29.097976    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh7wk\" (UniqueName: \"kubernetes.io/projected/724e2df0-ad34-4ddc-803f-c18987837f70-kube-api-access-nh7wk\") pod \"maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r\" (UID: \"724e2df0-ad34-4ddc-803f-c18987837f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:29.098663 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:29.098550    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r\" (UID: \"724e2df0-ad34-4ddc-803f-c18987837f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:29.100302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:29.100283    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r\" (UID: \"724e2df0-ad34-4ddc-803f-c18987837f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:29.112052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:29.112025    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh7wk\" (UniqueName: \"kubernetes.io/projected/724e2df0-ad34-4ddc-803f-c18987837f70-kube-api-access-nh7wk\") pod \"maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r\" (UID: \"724e2df0-ad34-4ddc-803f-c18987837f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:29.161792 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:29.161752    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:54:29.271414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:29.268883    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:29.468200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:29.468157    4918 generic.go:356] "Generic (PLEG): container finished" podID="652dc98f-92b3-4576-9efe-c5e5c521da9e" containerID="3ef5c9235aaf9e42eee27335e8788e136ccd74101bd2443c52349214ad029407" exitCode=0
Oct 30 21:54:29.468438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:29.468286    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl" event={"ID":"652dc98f-92b3-4576-9efe-c5e5c521da9e","Type":"ContainerDied","Data":"3ef5c9235aaf9e42eee27335e8788e136ccd74101bd2443c52349214ad029407"}
Oct 30 21:54:29.598289 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:29.598259    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724e2df0_ad34_4ddc_803f_c18987837f70.slice/crio-98d958b470a16f4cf5c5b4b6f8d04362497aa2842702e0bc92e5c9d462945960 WatchSource:0}: Error finding container 98d958b470a16f4cf5c5b4b6f8d04362497aa2842702e0bc92e5c9d462945960: Status 404 returned error can't find the container with id 98d958b470a16f4cf5c5b4b6f8d04362497aa2842702e0bc92e5c9d462945960
Oct 30 21:54:29.598997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:29.598970    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"]
Oct 30 21:54:30.215949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.215907    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc" podUID="3b022d69-5d42-42b2-bb5e-bbb98c4cedb8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:30.216484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.216229    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr" podUID="724e33b5-e637-47fb-b63f-dab247b15cd5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:30.479644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.479606    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r" event={"ID":"724e2df0-ad34-4ddc-803f-c18987837f70","Type":"ContainerStarted","Data":"f7e916439919725b7019521de826bdb2f13267f41012e90b88ddc4dc775c77d0"}
Oct 30 21:54:30.479815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.479675    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r" event={"ID":"724e2df0-ad34-4ddc-803f-c18987837f70","Type":"ContainerStarted","Data":"98d958b470a16f4cf5c5b4b6f8d04362497aa2842702e0bc92e5c9d462945960"}
Oct 30 21:54:30.511491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.511435    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r" podStartSLOduration=2.5113809959999998 podStartE2EDuration="2.511380996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:30.509249726 +0000 UTC m=+3183.555808872" watchObservedRunningTime="2026-10-30 21:54:30.511380996 +0000 UTC m=+3183.557940140"
Oct 30 21:54:30.653033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.652993    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:30.734925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.734904    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652dc98f-92b3-4576-9efe-c5e5c521da9e-kube-api-access-5k7tl\" (UniqueName: \"kubernetes.io/projected/652dc98f-92b3-4576-9efe-c5e5c521da9e-kube-api-access-5k7tl\") pod \"652dc98f-92b3-4576-9efe-c5e5c521da9e\" (UID: \"652dc98f-92b3-4576-9efe-c5e5c521da9e\") "
Oct 30 21:54:30.735062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.735051    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-openshift-ca\") pod \"652dc98f-92b3-4576-9efe-c5e5c521da9e\" (UID: \"652dc98f-92b3-4576-9efe-c5e5c521da9e\") "
Oct 30 21:54:30.735182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.735171    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-xks-ca\") pod \"652dc98f-92b3-4576-9efe-c5e5c521da9e\" (UID: \"652dc98f-92b3-4576-9efe-c5e5c521da9e\") "
Oct 30 21:54:30.735434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.735408    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-openshift-ca" pod "652dc98f-92b3-4576-9efe-c5e5c521da9e" (UID: "652dc98f-92b3-4576-9efe-c5e5c521da9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:30.735811 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.735792    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:30.737168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.737140    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652dc98f-92b3-4576-9efe-c5e5c521da9e-kube-api-access-5k7tl" pod "652dc98f-92b3-4576-9efe-c5e5c521da9e" (UID: "652dc98f-92b3-4576-9efe-c5e5c521da9e"). InnerVolumeSpecName "kube-api-access-5k7tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:30.737471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.737447    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-xks-ca" pod "652dc98f-92b3-4576-9efe-c5e5c521da9e" (UID: "652dc98f-92b3-4576-9efe-c5e5c521da9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:30.837066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.837037    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k7tl\" (UniqueName: \"kubernetes.io/projected/652dc98f-92b3-4576-9efe-c5e5c521da9e-kube-api-access-5k7tl\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:30.837066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:30.837066    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652dc98f-92b3-4576-9efe-c5e5c521da9e-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.490157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.490128    4918 generic.go:356] "Generic (PLEG): container finished" podID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerID="84d9f022521d2ef3b21f4d4119840f2fc525110c4b365619215477ed1c1d3be2" exitCode=0
Oct 30 21:54:31.490502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.490200    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" event={"ID":"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59","Type":"ContainerDied","Data":"84d9f022521d2ef3b21f4d4119840f2fc525110c4b365619215477ed1c1d3be2"}
Oct 30 21:54:31.490502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.490261    4918 scope.go:122] "RemoveContainer" containerID="acf0d574c3e6a38883f1b3712e0dd3bd6c8d5a308f896a051e7068e28d4ada50"
Oct 30 21:54:31.492098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.492078    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"
Oct 30 21:54:31.492207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.492098    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl" event={"ID":"652dc98f-92b3-4576-9efe-c5e5c521da9e","Type":"ContainerDied","Data":"7527b0a9626893b1f26add1c642f3174bdede1186ec56120e2bf66f4e887447d"}
Oct 30 21:54:31.492207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.492151    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7527b0a9626893b1f26add1c642f3174bdede1186ec56120e2bf66f4e887447d"
Oct 30 21:54:31.494060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.494032    4918 generic.go:356] "Generic (PLEG): container finished" podID="724e2df0-ad34-4ddc-803f-c18987837f70" containerID="f7e916439919725b7019521de826bdb2f13267f41012e90b88ddc4dc775c77d0" exitCode=0
Oct 30 21:54:31.494189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.494112    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r" event={"ID":"724e2df0-ad34-4ddc-803f-c18987837f70","Type":"ContainerDied","Data":"f7e916439919725b7019521de826bdb2f13267f41012e90b88ddc4dc775c77d0"}
Oct 30 21:54:31.670466 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.670441    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:54:31.753188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.753148    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-credential-socket\") pod \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") "
Oct 30 21:54:31.753440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.753269    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-token\" (UniqueName: \"kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-token\") pod \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") "
Oct 30 21:54:31.753534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.753437    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-data\") pod \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") "
Oct 30 21:54:31.753534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.753517    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-credential-socket" pod "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" (UID: "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:31.753658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.753597    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-podinfo\") pod \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") "
Oct 30 21:54:31.753658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.753643    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-socket\") pod \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") "
Oct 30 21:54:31.753785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.753689    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-certs\") pod \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") "
Oct 30 21:54:31.753785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.753751    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-kube-api-access-7wrlw\" (UniqueName: \"kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-kube-api-access-7wrlw\") pod \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") "
Oct 30 21:54:31.753912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.753683    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-data" pod "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" (UID: "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:31.753912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.753841    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-envoy\") pod \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") "
Oct 30 21:54:31.754026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.754000    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istiod-ca-cert\") pod \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\" (UID: \"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59\") "
Oct 30 21:54:31.754255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.754229    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-certs" pod "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" (UID: "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:31.754546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.754519    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istiod-ca-cert" pod "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" (UID: "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:31.754753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.754707    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-socket" pod "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" (UID: "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:31.754975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.754950    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.755078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.754976    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.755078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.754992    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.755078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.755005    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.755078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.755020    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.756786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.756711    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-podinfo" pod "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" (UID: "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:31.756786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.756720    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-envoy" pod "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" (UID: "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:31.757372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.757346    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-token" pod "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" (UID: "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:31.757504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.757430    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-kube-api-access-7wrlw" pod "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" (UID: "b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59"). InnerVolumeSpecName "kube-api-access-7wrlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:31.856019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.855981    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.856019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.856009    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.856019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.856020    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wrlw\" (UniqueName: \"kubernetes.io/projected/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-kube-api-access-7wrlw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.856359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:31.856030    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:32.508001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.507970    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"
Oct 30 21:54:32.508487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.508003    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4" event={"ID":"b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59","Type":"ContainerDied","Data":"3aaac4eaa6de48aa2bbf681434509d61aa1d6fb0359eb2f9e912be8a315d7ee7"}
Oct 30 21:54:32.508487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.508071    4918 scope.go:122] "RemoveContainer" containerID="84d9f022521d2ef3b21f4d4119840f2fc525110c4b365619215477ed1c1d3be2"
Oct 30 21:54:32.552223 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.552191    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"]
Oct 30 21:54:32.556454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.556424    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-87ae68-openshift-default-7b976f74ff-gdmh4"]
Oct 30 21:54:32.682186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.682161    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:32.771310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.771227    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-xks-ca\") pod \"724e2df0-ad34-4ddc-803f-c18987837f70\" (UID: \"724e2df0-ad34-4ddc-803f-c18987837f70\") "
Oct 30 21:54:32.771481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.771357    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-openshift-ca\") pod \"724e2df0-ad34-4ddc-803f-c18987837f70\" (UID: \"724e2df0-ad34-4ddc-803f-c18987837f70\") "
Oct 30 21:54:32.771481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.771469    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724e2df0-ad34-4ddc-803f-c18987837f70-kube-api-access-nh7wk\" (UniqueName: \"kubernetes.io/projected/724e2df0-ad34-4ddc-803f-c18987837f70-kube-api-access-nh7wk\") pod \"724e2df0-ad34-4ddc-803f-c18987837f70\" (UID: \"724e2df0-ad34-4ddc-803f-c18987837f70\") "
Oct 30 21:54:32.771732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.771705    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-openshift-ca" pod "724e2df0-ad34-4ddc-803f-c18987837f70" (UID: "724e2df0-ad34-4ddc-803f-c18987837f70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:32.772157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.772112    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:32.773616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.773593    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-xks-ca" pod "724e2df0-ad34-4ddc-803f-c18987837f70" (UID: "724e2df0-ad34-4ddc-803f-c18987837f70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:32.773856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.773841    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724e2df0-ad34-4ddc-803f-c18987837f70-kube-api-access-nh7wk" pod "724e2df0-ad34-4ddc-803f-c18987837f70" (UID: "724e2df0-ad34-4ddc-803f-c18987837f70"). InnerVolumeSpecName "kube-api-access-nh7wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:32.873933 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.873903    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh7wk\" (UniqueName: \"kubernetes.io/projected/724e2df0-ad34-4ddc-803f-c18987837f70-kube-api-access-nh7wk\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:32.873933 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:32.873934    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724e2df0-ad34-4ddc-803f-c18987837f70-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:33.081450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:33.081358    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" path="/var/lib/kubelet/pods/b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59/volumes"
Oct 30 21:54:33.519038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:33.518999    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"
Oct 30 21:54:33.519038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:33.519031    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r" event={"ID":"724e2df0-ad34-4ddc-803f-c18987837f70","Type":"ContainerDied","Data":"98d958b470a16f4cf5c5b4b6f8d04362497aa2842702e0bc92e5c9d462945960"}
Oct 30 21:54:33.519684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:33.519084    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98d958b470a16f4cf5c5b4b6f8d04362497aa2842702e0bc92e5c9d462945960"
Oct 30 21:54:34.079714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.078467    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"]
Oct 30 21:54:34.079714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.078683    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerName="istio-proxy" containerID="cri-o://9b8af7dfca8f02bf604c4d6af306d2fc260e4f7982bc7831f924e26af18030ea" gracePeriod=30
Oct 30 21:54:34.084018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.083990    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"]
Oct 30 21:54:34.085242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.085222    4918 cpu_manager.go:414] "containerMap: removing container" podUID="652dc98f-92b3-4576-9efe-c5e5c521da9e" containerName="revoke-keys"
Oct 30 21:54:34.085435 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.085374    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652dc98f-92b3-4576-9efe-c5e5c521da9e" containerName="revoke-keys"
Oct 30 21:54:34.085594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.085581    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy"
Oct 30 21:54:34.085677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.085667    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy"
Oct 30 21:54:34.085749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.085740    4918 cpu_manager.go:414] "containerMap: removing container" podUID="724e2df0-ad34-4ddc-803f-c18987837f70" containerName="revoke-keys"
Oct 30 21:54:34.085829 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.085819    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724e2df0-ad34-4ddc-803f-c18987837f70" containerName="revoke-keys"
Oct 30 21:54:34.085944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.085935    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy"
Oct 30 21:54:34.086017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.086008    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy"
Oct 30 21:54:34.086258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.086247    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy"
Oct 30 21:54:34.086338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.086330    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="724e2df0-ad34-4ddc-803f-c18987837f70" containerName="revoke-keys"
Oct 30 21:54:34.086441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.086431    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75e2a4e-7c27-4a3f-8cc0-09478b8dfa59" containerName="istio-proxy"
Oct 30 21:54:34.086529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.086520    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="652dc98f-92b3-4576-9efe-c5e5c521da9e" containerName="revoke-keys"
Oct 30 21:54:34.091486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.091456    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.099378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.099350    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e8d315-openshift-default-dockercfg-7c42t\""
Oct 30 21:54:34.102717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.102689    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"]
Oct 30 21:54:34.161365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.161322    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"]
Oct 30 21:54:34.165747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.165726    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f1ed97-tnf8t"]
Oct 30 21:54:34.241990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.241953    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-token\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.242182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.242100    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-podinfo\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.242182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.242156    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-socket\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.242305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.242214    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqcz2\" (UniqueName: \"kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-kube-api-access-bqcz2\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.242359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.242346    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-credential-socket\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.242547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.242520    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-data\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.242729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.242554    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istiod-ca-cert\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.242729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.242598    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-envoy\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.242851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.242759    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-certs\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.355959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.355852    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-token\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.356353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.356328    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-podinfo\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.362352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.357872    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-socket\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.362352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.357915    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqcz2\" (UniqueName: \"kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-kube-api-access-bqcz2\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.362352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.358007    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-credential-socket\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.362352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.358118    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-data\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.362352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.358144    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istiod-ca-cert\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.362352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.358190    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-envoy\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.362352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.358262    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-certs\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.362352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.358742    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-certs\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.362352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.359012    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-socket\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.362352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.359576    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-credential-socket\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.362352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.359815    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-data\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.362352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.360536    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istiod-ca-cert\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.363669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.363436    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-envoy\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.369452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.368019    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-podinfo\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.378665 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.378634    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-token\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.380325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.380272    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqcz2\" (UniqueName: \"kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-kube-api-access-bqcz2\") pod \"e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") " pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.421565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.421542    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:34.712932 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:34.712901    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c62c0a_7427_4c23_bee1_9d3a7cd9b2ba.slice/crio-919744ed21eee41eac609c0a586240f0014c8880deef7a3e07bd63049c36bfb1 WatchSource:0}: Error finding container 919744ed21eee41eac609c0a586240f0014c8880deef7a3e07bd63049c36bfb1: Status 404 returned error can't find the container with id 919744ed21eee41eac609c0a586240f0014c8880deef7a3e07bd63049c36bfb1
Oct 30 21:54:34.713371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.713348    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"]
Oct 30 21:54:34.715992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.715959    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:34.716085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.716024    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:34.716085 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.716054    4918 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:34.949436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.949354    4918 scope.go:122] "RemoveContainer" containerID="122ef3ba4778fe9d7a9579d730e3d9af322236bc4645f830bca5aa379e82edf4"
Oct 30 21:54:34.963687 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:34.963667    4918 scope.go:122] "RemoveContainer" containerID="c0ccec3337452b02d1f27f765aa2f38a30d928d77250cdf8a38305e3d1f9b1ef"
Oct 30 21:54:35.082453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:35.082428    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="879ea4a6-5d54-4546-ba61-6d6745e953ae" path="/var/lib/kubelet/pods/879ea4a6-5d54-4546-ba61-6d6745e953ae/volumes"
Oct 30 21:54:35.546110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:35.546075    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv" event={"ID":"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba","Type":"ContainerStarted","Data":"1bc4255c69d7772fd341b751783226ef3758b6bcc2a8bfe1b517380ac23bba61"}
Oct 30 21:54:35.546274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:35.546131    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv" event={"ID":"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba","Type":"ContainerStarted","Data":"919744ed21eee41eac609c0a586240f0014c8880deef7a3e07bd63049c36bfb1"}
Oct 30 21:54:35.619807 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:35.619747    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv" podStartSLOduration=1.619726819 podStartE2EDuration="1.619726819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:35.617881708 +0000 UTC m=+3188.664440861" watchObservedRunningTime="2026-10-30 21:54:35.619726819 +0000 UTC m=+3188.666285965"
Oct 30 21:54:36.422114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:36.422083    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:36.423194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:36.423172    4918 patch_prober.go:29] interesting pod/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" start-of-body=
Oct 30 21:54:36.423267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:36.423220    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv" podUID="a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused"
Oct 30 21:54:37.426381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:37.426282    4918 patch_prober.go:29] interesting pod/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" start-of-body=
Oct 30 21:54:37.426970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:37.426937    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv" podUID="a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused"
Oct 30 21:54:38.422198 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:38.422162    4918 patch_prober.go:29] interesting pod/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" start-of-body=
Oct 30 21:54:38.422400 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:38.422219    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv" podUID="a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused"
Oct 30 21:54:39.129618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:39.128986    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"]
Oct 30 21:54:39.426464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:39.426164    4918 patch_prober.go:29] interesting pod/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" start-of-body=
Oct 30 21:54:39.426464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:39.426224    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv" podUID="a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused"
Oct 30 21:54:39.450414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:39.444959    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"]
Oct 30 21:54:39.456049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:39.452669    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy" containerID="cri-o://fbf3d94c7bc7c81cbbfb617130f1fe77048d45bcacea4903644364e7258fd613" gracePeriod=30
Oct 30 21:54:39.664176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:39.664068    4918 generic.go:356] "Generic (PLEG): container finished" podID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerID="9b8af7dfca8f02bf604c4d6af306d2fc260e4f7982bc7831f924e26af18030ea" exitCode=0
Oct 30 21:54:39.664176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:39.664132    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" event={"ID":"82bc3cc6-e178-40d5-a7f0-240760e37408","Type":"ContainerDied","Data":"9b8af7dfca8f02bf604c4d6af306d2fc260e4f7982bc7831f924e26af18030ea"}
Oct 30 21:54:39.876981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:39.876955    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:54:40.038849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.038812    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-data\") pod \"82bc3cc6-e178-40d5-a7f0-240760e37408\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") "
Oct 30 21:54:40.039057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.038903    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-credential-socket\") pod \"82bc3cc6-e178-40d5-a7f0-240760e37408\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") "
Oct 30 21:54:40.039057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.038965    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-certs\") pod \"82bc3cc6-e178-40d5-a7f0-240760e37408\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") "
Oct 30 21:54:40.039057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.039018    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-envoy\") pod \"82bc3cc6-e178-40d5-a7f0-240760e37408\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") "
Oct 30 21:54:40.039057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.039052    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-token\" (UniqueName: \"kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-token\") pod \"82bc3cc6-e178-40d5-a7f0-240760e37408\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") "
Oct 30 21:54:40.039290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.039088    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-data" pod "82bc3cc6-e178-40d5-a7f0-240760e37408" (UID: "82bc3cc6-e178-40d5-a7f0-240760e37408"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:40.039290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.039146    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-podinfo\") pod \"82bc3cc6-e178-40d5-a7f0-240760e37408\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") "
Oct 30 21:54:40.039290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.039203    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-socket\") pod \"82bc3cc6-e178-40d5-a7f0-240760e37408\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") "
Oct 30 21:54:40.039290 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.039255    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-credential-socket" pod "82bc3cc6-e178-40d5-a7f0-240760e37408" (UID: "82bc3cc6-e178-40d5-a7f0-240760e37408"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:40.039543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.039352    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-certs" pod "82bc3cc6-e178-40d5-a7f0-240760e37408" (UID: "82bc3cc6-e178-40d5-a7f0-240760e37408"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:40.039543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.039499    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82bc3cc6-e178-40d5-a7f0-240760e37408-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82bc3cc6-e178-40d5-a7f0-240760e37408-istiod-ca-cert\") pod \"82bc3cc6-e178-40d5-a7f0-240760e37408\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") "
Oct 30 21:54:40.039668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.039543    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-socket" pod "82bc3cc6-e178-40d5-a7f0-240760e37408" (UID: "82bc3cc6-e178-40d5-a7f0-240760e37408"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:40.039734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.039668    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-kube-api-access-cpj4z\" (UniqueName: \"kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-kube-api-access-cpj4z\") pod \"82bc3cc6-e178-40d5-a7f0-240760e37408\" (UID: \"82bc3cc6-e178-40d5-a7f0-240760e37408\") "
Oct 30 21:54:40.039888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.039864    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82bc3cc6-e178-40d5-a7f0-240760e37408-istiod-ca-cert" pod "82bc3cc6-e178-40d5-a7f0-240760e37408" (UID: "82bc3cc6-e178-40d5-a7f0-240760e37408"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:40.040378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.040361    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82bc3cc6-e178-40d5-a7f0-240760e37408-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.040519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.040380    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.040519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.040430    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.040519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.040444    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.040519 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.040452    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.042056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.042031    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-podinfo" pod "82bc3cc6-e178-40d5-a7f0-240760e37408" (UID: "82bc3cc6-e178-40d5-a7f0-240760e37408"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:40.042278 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.042256    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-envoy" pod "82bc3cc6-e178-40d5-a7f0-240760e37408" (UID: "82bc3cc6-e178-40d5-a7f0-240760e37408"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:40.042513 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.042493    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-token" pod "82bc3cc6-e178-40d5-a7f0-240760e37408" (UID: "82bc3cc6-e178-40d5-a7f0-240760e37408"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:40.043338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.043317    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-kube-api-access-cpj4z" pod "82bc3cc6-e178-40d5-a7f0-240760e37408" (UID: "82bc3cc6-e178-40d5-a7f0-240760e37408"). InnerVolumeSpecName "kube-api-access-cpj4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:40.142345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.142267    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.142345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.142295    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.142345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.142308    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82bc3cc6-e178-40d5-a7f0-240760e37408-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.142345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.142319    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpj4z\" (UniqueName: \"kubernetes.io/projected/82bc3cc6-e178-40d5-a7f0-240760e37408-kube-api-access-cpj4z\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.211302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.211173    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc" podUID="3b022d69-5d42-42b2-bb5e-bbb98c4cedb8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:40.211701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.211672    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr" podUID="724e33b5-e637-47fb-b63f-dab247b15cd5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:40.423366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.423274    4918 patch_prober.go:29] interesting pod/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused" start-of-body=
Oct 30 21:54:40.423366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.423342    4918 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv" podUID="a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.169:15021/healthz/ready\": dial tcp 10.132.1.169:15021: connect: connection refused"
Oct 30 21:54:40.676800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.676715    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"
Oct 30 21:54:40.676800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.676744    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm" event={"ID":"82bc3cc6-e178-40d5-a7f0-240760e37408","Type":"ContainerDied","Data":"8137bc26ddbbddf5dfc21c2a727ef55cc483af25aa8b3f6e8dd91b8528e3870c"}
Oct 30 21:54:40.676999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.676823    4918 scope.go:122] "RemoveContainer" containerID="9b8af7dfca8f02bf604c4d6af306d2fc260e4f7982bc7831f924e26af18030ea"
Oct 30 21:54:40.735346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.735314    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"]
Oct 30 21:54:40.739300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:40.739275    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-h7txm"]
Oct 30 21:54:41.081855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:41.081799    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" path="/var/lib/kubelet/pods/82bc3cc6-e178-40d5-a7f0-240760e37408/volumes"
Oct 30 21:54:41.533674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:41.533645    4918 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:41.534351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:41.534272    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:41.535451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:41.535429    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:54:41.694827 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:41.694791    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"]
Oct 30 21:54:41.695084 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:41.695050    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy" containerID="cri-o://9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3" gracePeriod=30
Oct 30 21:54:41.711261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:41.711235    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"]
Oct 30 21:54:41.711508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:41.711409    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5" podUID="24c41b0e-4cdb-47e3-aa79-c5b22a123c62" containerName="istio-proxy" containerID="cri-o://c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a" gracePeriod=30
Oct 30 21:54:42.443181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.443143    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"]
Oct 30 21:54:42.449624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.449598    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a378a2-rslpl"]
Oct 30 21:54:42.496854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.496818    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:54:42.497013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.496884    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:54:42.507010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.506984    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"]
Oct 30 21:54:42.507735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.507720    4918 cpu_manager.go:414] "containerMap: removing container" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerName="istio-proxy"
Oct 30 21:54:42.507794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.507736    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerName="istio-proxy"
Oct 30 21:54:42.507882 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.507872    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="82bc3cc6-e178-40d5-a7f0-240760e37408" containerName="istio-proxy"
Oct 30 21:54:42.512569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.512552    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:42.518462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.518427    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:42.557636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.557606    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:54:42.586619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.586583    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"]
Oct 30 21:54:42.617186 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.617153    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7\" (UID: \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:42.617380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.617208    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z4k8\" (UniqueName: \"kubernetes.io/projected/05f0d8de-d487-4c5c-9222-f4bac0b23c49-kube-api-access-7z4k8\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7\" (UID: \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:42.617580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.617562    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7\" (UID: \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:42.718671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.718587    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7\" (UID: \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:42.718830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.718672    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7\" (UID: \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:42.718830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.718701    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4k8\" (UniqueName: \"kubernetes.io/projected/05f0d8de-d487-4c5c-9222-f4bac0b23c49-kube-api-access-7z4k8\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7\" (UID: \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:42.719330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.719304    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7\" (UID: \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:42.721174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.721146    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7\" (UID: \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:42.727815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.727785    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z4k8\" (UniqueName: \"kubernetes.io/projected/05f0d8de-d487-4c5c-9222-f4bac0b23c49-kube-api-access-7z4k8\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7\" (UID: \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:42.894240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:42.894209    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:43.082407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.082365    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652dc98f-92b3-4576-9efe-c5e5c521da9e" path="/var/lib/kubelet/pods/652dc98f-92b3-4576-9efe-c5e5c521da9e/volumes"
Oct 30 21:54:43.190229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.190200    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"]
Oct 30 21:54:43.191587 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:43.191561    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f0d8de_d487_4c5c_9222_f4bac0b23c49.slice/crio-5edbcd23c001ece4e204b14178a4ad7dc7564737f7d2d2520a4c1f28953da227 WatchSource:0}: Error finding container 5edbcd23c001ece4e204b14178a4ad7dc7564737f7d2d2520a4c1f28953da227: Status 404 returned error can't find the container with id 5edbcd23c001ece4e204b14178a4ad7dc7564737f7d2d2520a4c1f28953da227
Oct 30 21:54:43.709250 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.709215    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7" event={"ID":"05f0d8de-d487-4c5c-9222-f4bac0b23c49","Type":"ContainerStarted","Data":"fa9fc31303915b2b69373be5179577c456ba4a33d8401e83acbed949194a6361"}
Oct 30 21:54:43.709679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.709287    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7" event={"ID":"05f0d8de-d487-4c5c-9222-f4bac0b23c49","Type":"ContainerStarted","Data":"5edbcd23c001ece4e204b14178a4ad7dc7564737f7d2d2520a4c1f28953da227"}
Oct 30 21:54:43.711303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.711280    4918 generic.go:356] "Generic (PLEG): container finished" podID="724e33b5-e637-47fb-b63f-dab247b15cd5" containerID="2dea06b0a1433e40f2324ed25e7b8cf1acc515ec6b65e4603cefaa53990bdaee" exitCode=0
Oct 30 21:54:43.711463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.711351    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr" event={"ID":"724e33b5-e637-47fb-b63f-dab247b15cd5","Type":"ContainerDied","Data":"2dea06b0a1433e40f2324ed25e7b8cf1acc515ec6b65e4603cefaa53990bdaee"}
Oct 30 21:54:43.713351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.713320    4918 generic.go:356] "Generic (PLEG): container finished" podID="3b022d69-5d42-42b2-bb5e-bbb98c4cedb8" containerID="c0537919f97d123320641e8e998c93579c889a6a95438d483ccca70f141a603c" exitCode=0
Oct 30 21:54:43.713518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.713358    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc" event={"ID":"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8","Type":"ContainerDied","Data":"c0537919f97d123320641e8e998c93579c889a6a95438d483ccca70f141a603c"}
Oct 30 21:54:43.731240 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.731179    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7" podStartSLOduration=1.731159169 podStartE2EDuration="1.731159169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:43.729553545 +0000 UTC m=+3196.776112719" watchObservedRunningTime="2026-10-30 21:54:43.731159169 +0000 UTC m=+3196.777718315"
Oct 30 21:54:43.763010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.762980    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"]
Oct 30 21:54:43.763523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.763252    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv" podUID="de805cdc-648b-4f04-89e5-d39079fffd6b" containerName="discovery" containerID="cri-o://07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864" gracePeriod=30
Oct 30 21:54:43.924423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.921301    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"
Oct 30 21:54:43.969319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:43.969289    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"
Oct 30 21:54:44.046074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.045992    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724e33b5-e637-47fb-b63f-dab247b15cd5-kube-api-access-5db4h\" (UniqueName: \"kubernetes.io/projected/724e33b5-e637-47fb-b63f-dab247b15cd5-kube-api-access-5db4h\") pod \"724e33b5-e637-47fb-b63f-dab247b15cd5\" (UID: \"724e33b5-e637-47fb-b63f-dab247b15cd5\") "
Oct 30 21:54:44.046256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.046124    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724e33b5-e637-47fb-b63f-dab247b15cd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724e33b5-e637-47fb-b63f-dab247b15cd5-plugins-config-volume\") pod \"724e33b5-e637-47fb-b63f-dab247b15cd5\" (UID: \"724e33b5-e637-47fb-b63f-dab247b15cd5\") "
Oct 30 21:54:44.046973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.046919    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724e33b5-e637-47fb-b63f-dab247b15cd5-plugins-config-volume" pod "724e33b5-e637-47fb-b63f-dab247b15cd5" (UID: "724e33b5-e637-47fb-b63f-dab247b15cd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:44.047494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.047468    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/724e33b5-e637-47fb-b63f-dab247b15cd5-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:44.053655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.053617    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724e33b5-e637-47fb-b63f-dab247b15cd5-kube-api-access-5db4h" pod "724e33b5-e637-47fb-b63f-dab247b15cd5" (UID: "724e33b5-e637-47fb-b63f-dab247b15cd5"). InnerVolumeSpecName "kube-api-access-5db4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:44.122482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.122451    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:54:44.129357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.129326    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v" podUID="c1ef6ad5-f0ac-4096-8682-bfcbb76437a3" containerName="payload-pre-processing" containerID="cri-o://d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969" gracePeriod=25
Oct 30 21:54:44.148023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.147994    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-plugins-config-volume\") pod \"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8\" (UID: \"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8\") "
Oct 30 21:54:44.149187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.148339    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-kube-api-access-lb7ng\" (UniqueName: \"kubernetes.io/projected/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-kube-api-access-lb7ng\") pod \"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8\" (UID: \"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8\") "
Oct 30 21:54:44.149187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.148927    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-plugins-config-volume" pod "3b022d69-5d42-42b2-bb5e-bbb98c4cedb8" (UID: "3b022d69-5d42-42b2-bb5e-bbb98c4cedb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:44.149187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.149073    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:44.149600 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.149095    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5db4h\" (UniqueName: \"kubernetes.io/projected/724e33b5-e637-47fb-b63f-dab247b15cd5-kube-api-access-5db4h\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:44.153436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.152064    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-kube-api-access-lb7ng" pod "3b022d69-5d42-42b2-bb5e-bbb98c4cedb8" (UID: "3b022d69-5d42-42b2-bb5e-bbb98c4cedb8"). InnerVolumeSpecName "kube-api-access-lb7ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:44.250288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.250254    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-ca-configmap\") pod \"de805cdc-648b-4f04-89e5-d39079fffd6b\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") "
Oct 30 21:54:44.250529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.250488    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de805cdc-648b-4f04-89e5-d39079fffd6b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de805cdc-648b-4f04-89e5-d39079fffd6b-local-certs\") pod \"de805cdc-648b-4f04-89e5-d39079fffd6b\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") "
Oct 30 21:54:44.250529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.250522    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-dns-cert\") pod \"de805cdc-648b-4f04-89e5-d39079fffd6b\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") "
Oct 30 21:54:44.250885 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.250857    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-ca-configmap" pod "de805cdc-648b-4f04-89e5-d39079fffd6b" (UID: "de805cdc-648b-4f04-89e5-d39079fffd6b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:44.250975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.250919    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-cacerts\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-cacerts\") pod \"de805cdc-648b-4f04-89e5-d39079fffd6b\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") "
Oct 30 21:54:44.251015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.250981    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-token\" (UniqueName: \"kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-token\") pod \"de805cdc-648b-4f04-89e5-d39079fffd6b\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") "
Oct 30 21:54:44.251089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.251075    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-kubeconfig\") pod \"de805cdc-648b-4f04-89e5-d39079fffd6b\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") "
Oct 30 21:54:44.251142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.251126    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-kube-api-access-hzvwr\" (UniqueName: \"kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-kube-api-access-hzvwr\") pod \"de805cdc-648b-4f04-89e5-d39079fffd6b\" (UID: \"de805cdc-648b-4f04-89e5-d39079fffd6b\") "
Oct 30 21:54:44.252255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.252231    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb7ng\" (UniqueName: \"kubernetes.io/projected/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8-kube-api-access-lb7ng\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:44.252312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.252264    4918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-ca-configmap\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:44.253305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.253227    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-dns-cert" pod "de805cdc-648b-4f04-89e5-d39079fffd6b" (UID: "de805cdc-648b-4f04-89e5-d39079fffd6b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:44.256477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.256454    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-cacerts" pod "de805cdc-648b-4f04-89e5-d39079fffd6b" (UID: "de805cdc-648b-4f04-89e5-d39079fffd6b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:44.256675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.256647    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de805cdc-648b-4f04-89e5-d39079fffd6b-local-certs" pod "de805cdc-648b-4f04-89e5-d39079fffd6b" (UID: "de805cdc-648b-4f04-89e5-d39079fffd6b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:44.257985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.257958    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-token" pod "de805cdc-648b-4f04-89e5-d39079fffd6b" (UID: "de805cdc-648b-4f04-89e5-d39079fffd6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:44.258124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.258106    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-kubeconfig" pod "de805cdc-648b-4f04-89e5-d39079fffd6b" (UID: "de805cdc-648b-4f04-89e5-d39079fffd6b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:44.263740 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.263661    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-kube-api-access-hzvwr" pod "de805cdc-648b-4f04-89e5-d39079fffd6b" (UID: "de805cdc-648b-4f04-89e5-d39079fffd6b"). InnerVolumeSpecName "kube-api-access-hzvwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:44.353935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.353897    4918 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-kubeconfig\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:44.353935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.353938    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzvwr\" (UniqueName: \"kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-kube-api-access-hzvwr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:44.354175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.353953    4918 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de805cdc-648b-4f04-89e5-d39079fffd6b-local-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:44.354175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.353964    4918 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-csr-dns-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:44.354175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.353976    4918 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/de805cdc-648b-4f04-89e5-d39079fffd6b-cacerts\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:44.354175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.353988    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de805cdc-648b-4f04-89e5-d39079fffd6b-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:44.726087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.726055    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr" event={"ID":"724e33b5-e637-47fb-b63f-dab247b15cd5","Type":"ContainerDied","Data":"f8338e7c8540e018ca923b48b291b717bf24202ad9484160f204bce9620b27a4"}
Oct 30 21:54:44.726541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.726161    4918 scope.go:122] "RemoveContainer" containerID="2dea06b0a1433e40f2324ed25e7b8cf1acc515ec6b65e4603cefaa53990bdaee"
Oct 30 21:54:44.726541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.726316    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"
Oct 30 21:54:44.729179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.729139    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc" event={"ID":"3b022d69-5d42-42b2-bb5e-bbb98c4cedb8","Type":"ContainerDied","Data":"e9059fdd043ff641f60c1f16fe90b5238a95e88ec625515d2683f3e175c61f7e"}
Oct 30 21:54:44.729331 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.729293    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"
Oct 30 21:54:44.735430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.734626    4918 generic.go:356] "Generic (PLEG): container finished" podID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerID="fbf3d94c7bc7c81cbbfb617130f1fe77048d45bcacea4903644364e7258fd613" exitCode=0
Oct 30 21:54:44.735430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.735011    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" event={"ID":"0c84dfa1-93db-4289-85a3-0182e98b41cf","Type":"ContainerDied","Data":"fbf3d94c7bc7c81cbbfb617130f1fe77048d45bcacea4903644364e7258fd613"}
Oct 30 21:54:44.739246 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.738830    4918 generic.go:356] "Generic (PLEG): container finished" podID="05f0d8de-d487-4c5c-9222-f4bac0b23c49" containerID="fa9fc31303915b2b69373be5179577c456ba4a33d8401e83acbed949194a6361" exitCode=0
Oct 30 21:54:44.739427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.739266    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7" event={"ID":"05f0d8de-d487-4c5c-9222-f4bac0b23c49","Type":"ContainerDied","Data":"fa9fc31303915b2b69373be5179577c456ba4a33d8401e83acbed949194a6361"}
Oct 30 21:54:44.742265 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.742240    4918 generic.go:356] "Generic (PLEG): container finished" podID="de805cdc-648b-4f04-89e5-d39079fffd6b" containerID="07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864" exitCode=0
Oct 30 21:54:44.742381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.742294    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv" event={"ID":"de805cdc-648b-4f04-89e5-d39079fffd6b","Type":"ContainerDied","Data":"07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864"}
Oct 30 21:54:44.742381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.742335    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv" event={"ID":"de805cdc-648b-4f04-89e5-d39079fffd6b","Type":"ContainerDied","Data":"2dd1531b2c41910f5705b251b68118b4fba87f92011060d5b5c7bdefcef321da"}
Oct 30 21:54:44.749804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.749781    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"
Oct 30 21:54:44.764012 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.763993    4918 scope.go:122] "RemoveContainer" containerID="c0537919f97d123320641e8e998c93579c889a6a95438d483ccca70f141a603c"
Oct 30 21:54:44.774380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.774358    4918 scope.go:122] "RemoveContainer" containerID="54b994aa348f3ddd9bc2ba22b8ec7752ed568f0245513d43d7f990f49fd89d5a"
Oct 30 21:54:44.789836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.789817    4918 scope.go:122] "RemoveContainer" containerID="07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864"
Oct 30 21:54:44.802309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.802285    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"]
Oct 30 21:54:44.805265 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.805235    4918 scope.go:122] "RemoveContainer" containerID="07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864"
Oct 30 21:54:44.805618 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:44.805594    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864\": container with ID starting with 07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864 not found: ID does not exist" containerID="07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864"
Oct 30 21:54:44.805711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.805626    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864"} err="failed to get container status \"07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864\": rpc error: code = NotFound desc = could not find container \"07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864\": container with ID starting with 07d0f79950263ac31fba7fc408a445c269012e1631de7b51975bac45b0ffe864 not found: ID does not exist"
Oct 30 21:54:44.812145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.812122    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1ed97-648f56494f-qxxfr"]
Oct 30 21:54:44.842164 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.842137    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"]
Oct 30 21:54:44.846582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.846557    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f1ed97-6b55986c99-j6scc"]
Oct 30 21:54:44.860573 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.860533    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"]
Oct 30 21:54:44.865764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:44.865743    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hvrwv"]
Oct 30 21:54:45.081860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.081829    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b022d69-5d42-42b2-bb5e-bbb98c4cedb8" path="/var/lib/kubelet/pods/3b022d69-5d42-42b2-bb5e-bbb98c4cedb8/volumes"
Oct 30 21:54:45.082270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.082256    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724e33b5-e637-47fb-b63f-dab247b15cd5" path="/var/lib/kubelet/pods/724e33b5-e637-47fb-b63f-dab247b15cd5/volumes"
Oct 30 21:54:45.082730 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.082716    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de805cdc-648b-4f04-89e5-d39079fffd6b" path="/var/lib/kubelet/pods/de805cdc-648b-4f04-89e5-d39079fffd6b/volumes"
Oct 30 21:54:45.361423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.361380    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:54:45.471204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471174    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-envoy\") pod \"0c84dfa1-93db-4289-85a3-0182e98b41cf\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") "
Oct 30 21:54:45.471368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471214    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-certs\") pod \"0c84dfa1-93db-4289-85a3-0182e98b41cf\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") "
Oct 30 21:54:45.471368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471256    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c84dfa1-93db-4289-85a3-0182e98b41cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c84dfa1-93db-4289-85a3-0182e98b41cf-istiod-ca-cert\") pod \"0c84dfa1-93db-4289-85a3-0182e98b41cf\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") "
Oct 30 21:54:45.471368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471300    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-socket\") pod \"0c84dfa1-93db-4289-85a3-0182e98b41cf\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") "
Oct 30 21:54:45.471368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471333    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-data\") pod \"0c84dfa1-93db-4289-85a3-0182e98b41cf\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") "
Oct 30 21:54:45.471650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471414    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-credential-socket\") pod \"0c84dfa1-93db-4289-85a3-0182e98b41cf\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") "
Oct 30 21:54:45.471650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471582    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-certs" pod "0c84dfa1-93db-4289-85a3-0182e98b41cf" (UID: "0c84dfa1-93db-4289-85a3-0182e98b41cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:45.471650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471593    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-token\" (UniqueName: \"kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-token\") pod \"0c84dfa1-93db-4289-85a3-0182e98b41cf\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") "
Oct 30 21:54:45.471812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471715    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-data" pod "0c84dfa1-93db-4289-85a3-0182e98b41cf" (UID: "0c84dfa1-93db-4289-85a3-0182e98b41cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:45.471812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471734    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-socket" pod "0c84dfa1-93db-4289-85a3-0182e98b41cf" (UID: "0c84dfa1-93db-4289-85a3-0182e98b41cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:45.471812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471748    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-credential-socket" pod "0c84dfa1-93db-4289-85a3-0182e98b41cf" (UID: "0c84dfa1-93db-4289-85a3-0182e98b41cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:45.471812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471753    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-kube-api-access-vc69c\" (UniqueName: \"kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-kube-api-access-vc69c\") pod \"0c84dfa1-93db-4289-85a3-0182e98b41cf\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") "
Oct 30 21:54:45.472018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471834    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-podinfo\") pod \"0c84dfa1-93db-4289-85a3-0182e98b41cf\" (UID: \"0c84dfa1-93db-4289-85a3-0182e98b41cf\") "
Oct 30 21:54:45.472018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.471854    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c84dfa1-93db-4289-85a3-0182e98b41cf-istiod-ca-cert" pod "0c84dfa1-93db-4289-85a3-0182e98b41cf" (UID: "0c84dfa1-93db-4289-85a3-0182e98b41cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:45.472528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.472503    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:45.472528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.472524    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c84dfa1-93db-4289-85a3-0182e98b41cf-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:45.472723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.472533    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:45.472723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.472542    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:45.472723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.472557    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:45.473836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.473802    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-envoy" pod "0c84dfa1-93db-4289-85a3-0182e98b41cf" (UID: "0c84dfa1-93db-4289-85a3-0182e98b41cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:45.474133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.474112    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-kube-api-access-vc69c" pod "0c84dfa1-93db-4289-85a3-0182e98b41cf" (UID: "0c84dfa1-93db-4289-85a3-0182e98b41cf"). InnerVolumeSpecName "kube-api-access-vc69c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:45.474382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.474366    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-token" pod "0c84dfa1-93db-4289-85a3-0182e98b41cf" (UID: "0c84dfa1-93db-4289-85a3-0182e98b41cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:45.474382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.474369    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-podinfo" pod "0c84dfa1-93db-4289-85a3-0182e98b41cf" (UID: "0c84dfa1-93db-4289-85a3-0182e98b41cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:45.573791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.573760    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:45.573791 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.573789    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc69c\" (UniqueName: \"kubernetes.io/projected/0c84dfa1-93db-4289-85a3-0182e98b41cf-kube-api-access-vc69c\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:45.574013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.573799    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:45.574013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.573816    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c84dfa1-93db-4289-85a3-0182e98b41cf-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:45.753766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.753733    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"
Oct 30 21:54:45.754242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.753780    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw" event={"ID":"0c84dfa1-93db-4289-85a3-0182e98b41cf","Type":"ContainerDied","Data":"6e7ee2c22b5dc489d6ab152052e18d8e46a006c4e458916e7e340182b4a2cc82"}
Oct 30 21:54:45.754242 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.753859    4918 scope.go:122] "RemoveContainer" containerID="fbf3d94c7bc7c81cbbfb617130f1fe77048d45bcacea4903644364e7258fd613"
Oct 30 21:54:45.768614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.768589    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"]
Oct 30 21:54:45.769524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769506    4918 cpu_manager.go:414] "containerMap: removing container" podUID="3b022d69-5d42-42b2-bb5e-bbb98c4cedb8" containerName="payload-pre-processing"
Oct 30 21:54:45.769622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769524    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b022d69-5d42-42b2-bb5e-bbb98c4cedb8" containerName="payload-pre-processing"
Oct 30 21:54:45.769622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769544    4918 cpu_manager.go:414] "containerMap: removing container" podUID="de805cdc-648b-4f04-89e5-d39079fffd6b" containerName="discovery"
Oct 30 21:54:45.769622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769550    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de805cdc-648b-4f04-89e5-d39079fffd6b" containerName="discovery"
Oct 30 21:54:45.769622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769560    4918 cpu_manager.go:414] "containerMap: removing container" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy"
Oct 30 21:54:45.769622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769565    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy"
Oct 30 21:54:45.769622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769601    4918 cpu_manager.go:414] "containerMap: removing container" podUID="724e33b5-e637-47fb-b63f-dab247b15cd5" containerName="payload-processing"
Oct 30 21:54:45.769622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769606    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724e33b5-e637-47fb-b63f-dab247b15cd5" containerName="payload-processing"
Oct 30 21:54:45.769861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769706    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="de805cdc-648b-4f04-89e5-d39079fffd6b" containerName="discovery"
Oct 30 21:54:45.769861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769715    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy"
Oct 30 21:54:45.769861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769727    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy"
Oct 30 21:54:45.769861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769734    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="724e33b5-e637-47fb-b63f-dab247b15cd5" containerName="payload-processing"
Oct 30 21:54:45.769861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.769744    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b022d69-5d42-42b2-bb5e-bbb98c4cedb8" containerName="payload-pre-processing"
Oct 30 21:54:45.774580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.774560    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:45.780174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.779997    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"]
Oct 30 21:54:45.787551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.785428    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"]
Oct 30 21:54:45.792053 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.792026    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-w7ncw"]
Oct 30 21:54:45.880888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.880861    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf82b429-vjqtw\" (UID: \"e258e5ec-224a-4747-9880-0b09bb19265a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:45.881853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.881822    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5bth\" (UniqueName: \"kubernetes.io/projected/e258e5ec-224a-4747-9880-0b09bb19265a-kube-api-access-c5bth\") pod \"maas-api-revoke-sub-cf82b429-vjqtw\" (UID: \"e258e5ec-224a-4747-9880-0b09bb19265a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:45.882689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.882668    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf82b429-vjqtw\" (UID: \"e258e5ec-224a-4747-9880-0b09bb19265a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:45.951580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.951554    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:45.995372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.995328    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf82b429-vjqtw\" (UID: \"e258e5ec-224a-4747-9880-0b09bb19265a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:45.997102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.995709    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf82b429-vjqtw\" (UID: \"e258e5ec-224a-4747-9880-0b09bb19265a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:45.997566 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.997541    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5bth\" (UniqueName: \"kubernetes.io/projected/e258e5ec-224a-4747-9880-0b09bb19265a-kube-api-access-c5bth\") pod \"maas-api-revoke-sub-cf82b429-vjqtw\" (UID: \"e258e5ec-224a-4747-9880-0b09bb19265a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:45.998917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:45.998892    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf82b429-vjqtw\" (UID: \"e258e5ec-224a-4747-9880-0b09bb19265a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:46.011471 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.009508    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf82b429-vjqtw\" (UID: \"e258e5ec-224a-4747-9880-0b09bb19265a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:46.031491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.031460    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5bth\" (UniqueName: \"kubernetes.io/projected/e258e5ec-224a-4747-9880-0b09bb19265a-kube-api-access-c5bth\") pod \"maas-api-revoke-sub-cf82b429-vjqtw\" (UID: \"e258e5ec-224a-4747-9880-0b09bb19265a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:46.098066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.098036    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:46.098463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.098445    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-xks-ca\") pod \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\" (UID: \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\") "
Oct 30 21:54:46.098527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.098499    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f0d8de-d487-4c5c-9222-f4bac0b23c49-kube-api-access-7z4k8\" (UniqueName: \"kubernetes.io/projected/05f0d8de-d487-4c5c-9222-f4bac0b23c49-kube-api-access-7z4k8\") pod \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\" (UID: \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\") "
Oct 30 21:54:46.098607 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.098596    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-openshift-ca\") pod \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\" (UID: \"05f0d8de-d487-4c5c-9222-f4bac0b23c49\") "
Oct 30 21:54:46.098972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.098947    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-openshift-ca" pod "05f0d8de-d487-4c5c-9222-f4bac0b23c49" (UID: "05f0d8de-d487-4c5c-9222-f4bac0b23c49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:46.099258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.099232    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:46.100746 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.100723    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-xks-ca" pod "05f0d8de-d487-4c5c-9222-f4bac0b23c49" (UID: "05f0d8de-d487-4c5c-9222-f4bac0b23c49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:46.100984 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.100969    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f0d8de-d487-4c5c-9222-f4bac0b23c49-kube-api-access-7z4k8" pod "05f0d8de-d487-4c5c-9222-f4bac0b23c49" (UID: "05f0d8de-d487-4c5c-9222-f4bac0b23c49"). InnerVolumeSpecName "kube-api-access-7z4k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:46.202083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.202053    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05f0d8de-d487-4c5c-9222-f4bac0b23c49-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:46.202083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.202082    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z4k8\" (UniqueName: \"kubernetes.io/projected/05f0d8de-d487-4c5c-9222-f4bac0b23c49-kube-api-access-7z4k8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:46.387512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.387485    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"]
Oct 30 21:54:46.388277 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:46.388252    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode258e5ec_224a_4747_9880_0b09bb19265a.slice/crio-af2f23e1ac3f85cc0570df36c930fb5e060672f16c79a3d921b8f6bc21a4c64f WatchSource:0}: Error finding container af2f23e1ac3f85cc0570df36c930fb5e060672f16c79a3d921b8f6bc21a4c64f: Status 404 returned error can't find the container with id af2f23e1ac3f85cc0570df36c930fb5e060672f16c79a3d921b8f6bc21a4c64f
Oct 30 21:54:46.638894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.638814    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:54:46.638894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.638874    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5" podUID="24c41b0e-4cdb-47e3-aa79-c5b22a123c62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:54:46.767214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.767186    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"
Oct 30 21:54:46.767820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.767260    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7" event={"ID":"05f0d8de-d487-4c5c-9222-f4bac0b23c49","Type":"ContainerDied","Data":"5edbcd23c001ece4e204b14178a4ad7dc7564737f7d2d2520a4c1f28953da227"}
Oct 30 21:54:46.767820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.767318    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5edbcd23c001ece4e204b14178a4ad7dc7564737f7d2d2520a4c1f28953da227"
Oct 30 21:54:46.769434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:46.769376    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw" event={"ID":"e258e5ec-224a-4747-9880-0b09bb19265a","Type":"ContainerStarted","Data":"af2f23e1ac3f85cc0570df36c930fb5e060672f16c79a3d921b8f6bc21a4c64f"}
Oct 30 21:54:47.081207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.081173    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" path="/var/lib/kubelet/pods/0c84dfa1-93db-4289-85a3-0182e98b41cf/volumes"
Oct 30 21:54:47.137421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.136539    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"]
Oct 30 21:54:47.142828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.142803    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-36a099-ls97r"]
Oct 30 21:54:47.237794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.237766    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:54:47.276658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.276617    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28d7835c-782d-400b-8c1e-3a4654fd8a12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d7835c-782d-400b-8c1e-3a4654fd8a12-istiod-ca-cert\") pod \"28d7835c-782d-400b-8c1e-3a4654fd8a12\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") "
Oct 30 21:54:47.276837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.276715    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-kube-api-access-5sx76\" (UniqueName: \"kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-kube-api-access-5sx76\") pod \"28d7835c-782d-400b-8c1e-3a4654fd8a12\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") "
Oct 30 21:54:47.276837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.276756    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-envoy\") pod \"28d7835c-782d-400b-8c1e-3a4654fd8a12\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") "
Oct 30 21:54:47.276951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.276841    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-socket\") pod \"28d7835c-782d-400b-8c1e-3a4654fd8a12\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") "
Oct 30 21:54:47.276951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.276881    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-data\") pod \"28d7835c-782d-400b-8c1e-3a4654fd8a12\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") "
Oct 30 21:54:47.277023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.276986    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-token\" (UniqueName: \"kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-token\") pod \"28d7835c-782d-400b-8c1e-3a4654fd8a12\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") "
Oct 30 21:54:47.277078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.277053    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d7835c-782d-400b-8c1e-3a4654fd8a12-istiod-ca-cert" pod "28d7835c-782d-400b-8c1e-3a4654fd8a12" (UID: "28d7835c-782d-400b-8c1e-3a4654fd8a12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:47.277139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.277070    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-credential-socket\") pod \"28d7835c-782d-400b-8c1e-3a4654fd8a12\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") "
Oct 30 21:54:47.277139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.277115    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-certs\") pod \"28d7835c-782d-400b-8c1e-3a4654fd8a12\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") "
Oct 30 21:54:47.277222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.277196    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-podinfo\") pod \"28d7835c-782d-400b-8c1e-3a4654fd8a12\" (UID: \"28d7835c-782d-400b-8c1e-3a4654fd8a12\") "
Oct 30 21:54:47.277353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.277332    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-socket" pod "28d7835c-782d-400b-8c1e-3a4654fd8a12" (UID: "28d7835c-782d-400b-8c1e-3a4654fd8a12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.277994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.277959    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d7835c-782d-400b-8c1e-3a4654fd8a12-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.277994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.277988    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.278270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.278044    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-data" pod "28d7835c-782d-400b-8c1e-3a4654fd8a12" (UID: "28d7835c-782d-400b-8c1e-3a4654fd8a12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.278270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.278147    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-certs" pod "28d7835c-782d-400b-8c1e-3a4654fd8a12" (UID: "28d7835c-782d-400b-8c1e-3a4654fd8a12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.278537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.278494    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-credential-socket" pod "28d7835c-782d-400b-8c1e-3a4654fd8a12" (UID: "28d7835c-782d-400b-8c1e-3a4654fd8a12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.280439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.279912    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-token" pod "28d7835c-782d-400b-8c1e-3a4654fd8a12" (UID: "28d7835c-782d-400b-8c1e-3a4654fd8a12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:47.280439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.280105    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-podinfo" pod "28d7835c-782d-400b-8c1e-3a4654fd8a12" (UID: "28d7835c-782d-400b-8c1e-3a4654fd8a12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:47.280439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.280315    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-kube-api-access-5sx76" pod "28d7835c-782d-400b-8c1e-3a4654fd8a12" (UID: "28d7835c-782d-400b-8c1e-3a4654fd8a12"). InnerVolumeSpecName "kube-api-access-5sx76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:47.280439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.280369    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-envoy" pod "28d7835c-782d-400b-8c1e-3a4654fd8a12" (UID: "28d7835c-782d-400b-8c1e-3a4654fd8a12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.357418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.354134    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:54:47.379625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.379594    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sx76\" (UniqueName: \"kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-kube-api-access-5sx76\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.379625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.379622    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.379625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.379633    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.379948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.379642    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.379948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.379651    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.379948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.379659    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d7835c-782d-400b-8c1e-3a4654fd8a12-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.379948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.379667    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d7835c-782d-400b-8c1e-3a4654fd8a12-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.481071 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481032    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-envoy\") pod \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") "
Oct 30 21:54:47.481255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481101    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-certs\") pod \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") "
Oct 30 21:54:47.481255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481145    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-podinfo\") pod \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") "
Oct 30 21:54:47.481255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481222    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-kube-api-access-pfq9f\" (UniqueName: \"kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-kube-api-access-pfq9f\") pod \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") "
Oct 30 21:54:47.481520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481293    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istiod-ca-cert\") pod \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") "
Oct 30 21:54:47.481520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481370    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-socket\") pod \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") "
Oct 30 21:54:47.481520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481446    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-certs" pod "24c41b0e-4cdb-47e3-aa79-c5b22a123c62" (UID: "24c41b0e-4cdb-47e3-aa79-c5b22a123c62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.481520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481495    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-credential-socket\") pod \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") "
Oct 30 21:54:47.481734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481608    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-token\" (UniqueName: \"kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-token\") pod \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") "
Oct 30 21:54:47.481734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481645    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-data\") pod \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\" (UID: \"24c41b0e-4cdb-47e3-aa79-c5b22a123c62\") "
Oct 30 21:54:47.481846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481803    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istiod-ca-cert" pod "24c41b0e-4cdb-47e3-aa79-c5b22a123c62" (UID: "24c41b0e-4cdb-47e3-aa79-c5b22a123c62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:47.481974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481951    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-socket" pod "24c41b0e-4cdb-47e3-aa79-c5b22a123c62" (UID: "24c41b0e-4cdb-47e3-aa79-c5b22a123c62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.482063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.481971    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-credential-socket" pod "24c41b0e-4cdb-47e3-aa79-c5b22a123c62" (UID: "24c41b0e-4cdb-47e3-aa79-c5b22a123c62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.482206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.482179    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-data" pod "24c41b0e-4cdb-47e3-aa79-c5b22a123c62" (UID: "24c41b0e-4cdb-47e3-aa79-c5b22a123c62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.482366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.482351    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.482366 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.482366    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.482544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.482375    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.482544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.482404    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.482544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.482417    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.483553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.483534    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-envoy" pod "24c41b0e-4cdb-47e3-aa79-c5b22a123c62" (UID: "24c41b0e-4cdb-47e3-aa79-c5b22a123c62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:47.483821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.483807    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-token" pod "24c41b0e-4cdb-47e3-aa79-c5b22a123c62" (UID: "24c41b0e-4cdb-47e3-aa79-c5b22a123c62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:47.484112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.484091    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-podinfo" pod "24c41b0e-4cdb-47e3-aa79-c5b22a123c62" (UID: "24c41b0e-4cdb-47e3-aa79-c5b22a123c62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:47.484179 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.484110    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-kube-api-access-pfq9f" pod "24c41b0e-4cdb-47e3-aa79-c5b22a123c62" (UID: "24c41b0e-4cdb-47e3-aa79-c5b22a123c62"). InnerVolumeSpecName "kube-api-access-pfq9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:47.583840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.583802    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.583840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.583834    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.583840 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.583845    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfq9f\" (UniqueName: \"kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-kube-api-access-pfq9f\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.584109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.583855    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c41b0e-4cdb-47e3-aa79-c5b22a123c62-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:47.779948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.779856    4918 generic.go:356] "Generic (PLEG): container finished" podID="e258e5ec-224a-4747-9880-0b09bb19265a" containerID="87be476f8c6ebeb7f12f20d3c3df7606a62713c8d95c1544c69b92fe93020eda" exitCode=0
Oct 30 21:54:47.779948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.779932    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw" event={"ID":"e258e5ec-224a-4747-9880-0b09bb19265a","Type":"ContainerDied","Data":"87be476f8c6ebeb7f12f20d3c3df7606a62713c8d95c1544c69b92fe93020eda"}
Oct 30 21:54:47.781505 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.781479    4918 generic.go:356] "Generic (PLEG): container finished" podID="24c41b0e-4cdb-47e3-aa79-c5b22a123c62" containerID="c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a" exitCode=0
Oct 30 21:54:47.781617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.781523    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"
Oct 30 21:54:47.781617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.781562    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5" event={"ID":"24c41b0e-4cdb-47e3-aa79-c5b22a123c62","Type":"ContainerDied","Data":"c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a"}
Oct 30 21:54:47.781617 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.781609    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5" event={"ID":"24c41b0e-4cdb-47e3-aa79-c5b22a123c62","Type":"ContainerDied","Data":"34a5c27276e1e0186635af3931a44c9709f8edee7682d5979c86a940ad01a5ee"}
Oct 30 21:54:47.781750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.781640    4918 scope.go:122] "RemoveContainer" containerID="c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a"
Oct 30 21:54:47.783625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.783603    4918 generic.go:356] "Generic (PLEG): container finished" podID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerID="9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3" exitCode=0
Oct 30 21:54:47.783720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.783647    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"
Oct 30 21:54:47.783770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.783647    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" event={"ID":"28d7835c-782d-400b-8c1e-3a4654fd8a12","Type":"ContainerDied","Data":"9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3"}
Oct 30 21:54:47.783812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.783787    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl" event={"ID":"28d7835c-782d-400b-8c1e-3a4654fd8a12","Type":"ContainerDied","Data":"d7118edf83c51d91e6effe8db4e66b1be9018c9495662186028bc46dd221bdbf"}
Oct 30 21:54:47.794855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.794833    4918 scope.go:122] "RemoveContainer" containerID="c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a"
Oct 30 21:54:47.795266 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:47.795232    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a\": container with ID starting with c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a not found: ID does not exist" containerID="c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a"
Oct 30 21:54:47.795356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.795269    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a"} err="failed to get container status \"c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a\": rpc error: code = NotFound desc = could not find container \"c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a\": container with ID starting with c2267a56380477836693a36993affd4d3b6903ad1a2a79398ecc0e2df4c52d1a not found: ID does not exist"
Oct 30 21:54:47.795356 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.795295    4918 scope.go:122] "RemoveContainer" containerID="9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3"
Oct 30 21:54:47.890846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.890816    4918 scope.go:122] "RemoveContainer" containerID="fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317"
Oct 30 21:54:47.971410 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.964962    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"]
Oct 30 21:54:47.989411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:47.987444    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a378a2-openshift-default-5b788d5d5-qbql5"]
Oct 30 21:54:48.002468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:48.002332    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"]
Oct 30 21:54:48.009060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:48.009027    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f1ed97-openshift-default-6bd85645bf-rwvgl"]
Oct 30 21:54:48.019661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:48.019530    4918 scope.go:122] "RemoveContainer" containerID="9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3"
Oct 30 21:54:48.020021 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:48.019981    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3\": container with ID starting with 9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3 not found: ID does not exist" containerID="9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3"
Oct 30 21:54:48.020134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:48.020029    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3"} err="failed to get container status \"9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3\": rpc error: code = NotFound desc = could not find container \"9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3\": container with ID starting with 9977a154bbbad53ccd6f398aea8168c4e352350c7a8cc2446aaaaa9581256cb3 not found: ID does not exist"
Oct 30 21:54:48.020134 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:48.020050    4918 scope.go:122] "RemoveContainer" containerID="fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317"
Oct 30 21:54:48.020679 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:48.020368    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317\": container with ID starting with fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317 not found: ID does not exist" containerID="fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317"
Oct 30 21:54:48.020679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:48.020411    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317"} err="failed to get container status \"fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317\": rpc error: code = NotFound desc = could not find container \"fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317\": container with ID starting with fde92205c6471255809da45458efad4ddc4da4e1e4c228bd973bccd4a7959317 not found: ID does not exist"
Oct 30 21:54:48.959636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:48.959614    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:49.016260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.016229    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-xks-ca\") pod \"e258e5ec-224a-4747-9880-0b09bb19265a\" (UID: \"e258e5ec-224a-4747-9880-0b09bb19265a\") "
Oct 30 21:54:49.016487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.016277    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e258e5ec-224a-4747-9880-0b09bb19265a-kube-api-access-c5bth\" (UniqueName: \"kubernetes.io/projected/e258e5ec-224a-4747-9880-0b09bb19265a-kube-api-access-c5bth\") pod \"e258e5ec-224a-4747-9880-0b09bb19265a\" (UID: \"e258e5ec-224a-4747-9880-0b09bb19265a\") "
Oct 30 21:54:49.016487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.016352    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-openshift-ca\") pod \"e258e5ec-224a-4747-9880-0b09bb19265a\" (UID: \"e258e5ec-224a-4747-9880-0b09bb19265a\") "
Oct 30 21:54:49.017058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.017025    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-openshift-ca" pod "e258e5ec-224a-4747-9880-0b09bb19265a" (UID: "e258e5ec-224a-4747-9880-0b09bb19265a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:49.017273 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.017247    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:49.019045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.019020    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e258e5ec-224a-4747-9880-0b09bb19265a-kube-api-access-c5bth" pod "e258e5ec-224a-4747-9880-0b09bb19265a" (UID: "e258e5ec-224a-4747-9880-0b09bb19265a"). InnerVolumeSpecName "kube-api-access-c5bth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:49.019284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.019260    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-xks-ca" pod "e258e5ec-224a-4747-9880-0b09bb19265a" (UID: "e258e5ec-224a-4747-9880-0b09bb19265a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:49.079502 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.079420    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c41b0e-4cdb-47e3-aa79-c5b22a123c62" path="/var/lib/kubelet/pods/24c41b0e-4cdb-47e3-aa79-c5b22a123c62/volumes"
Oct 30 21:54:49.080060 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.080040    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" path="/var/lib/kubelet/pods/28d7835c-782d-400b-8c1e-3a4654fd8a12/volumes"
Oct 30 21:54:49.080531 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.080516    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724e2df0-ad34-4ddc-803f-c18987837f70" path="/var/lib/kubelet/pods/724e2df0-ad34-4ddc-803f-c18987837f70/volumes"
Oct 30 21:54:49.118946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.118919    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e258e5ec-224a-4747-9880-0b09bb19265a-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:49.118946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.118948    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5bth\" (UniqueName: \"kubernetes.io/projected/e258e5ec-224a-4747-9880-0b09bb19265a-kube-api-access-c5bth\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:49.803557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.803530    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw"
Oct 30 21:54:49.803714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.803565    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf82b429-vjqtw" event={"ID":"e258e5ec-224a-4747-9880-0b09bb19265a","Type":"ContainerDied","Data":"af2f23e1ac3f85cc0570df36c930fb5e060672f16c79a3d921b8f6bc21a4c64f"}
Oct 30 21:54:49.803714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:49.803613    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af2f23e1ac3f85cc0570df36c930fb5e060672f16c79a3d921b8f6bc21a4c64f"
Oct 30 21:54:51.246586 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.246547    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"]
Oct 30 21:54:51.247437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247415    4918 cpu_manager.go:414] "containerMap: removing container" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy"
Oct 30 21:54:51.247437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247437    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c84dfa1-93db-4289-85a3-0182e98b41cf" containerName="istio-proxy"
Oct 30 21:54:51.247604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247458    4918 cpu_manager.go:414] "containerMap: removing container" podUID="05f0d8de-d487-4c5c-9222-f4bac0b23c49" containerName="revoke-keys"
Oct 30 21:54:51.247604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247465    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f0d8de-d487-4c5c-9222-f4bac0b23c49" containerName="revoke-keys"
Oct 30 21:54:51.247604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247539    4918 cpu_manager.go:414] "containerMap: removing container" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy"
Oct 30 21:54:51.247604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247547    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy"
Oct 30 21:54:51.247604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247568    4918 cpu_manager.go:414] "containerMap: removing container" podUID="24c41b0e-4cdb-47e3-aa79-c5b22a123c62" containerName="istio-proxy"
Oct 30 21:54:51.247604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247576    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c41b0e-4cdb-47e3-aa79-c5b22a123c62" containerName="istio-proxy"
Oct 30 21:54:51.247919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247611    4918 cpu_manager.go:414] "containerMap: removing container" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy"
Oct 30 21:54:51.247919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247618    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy"
Oct 30 21:54:51.247919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247631    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e258e5ec-224a-4747-9880-0b09bb19265a" containerName="revoke-keys"
Oct 30 21:54:51.247919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247637    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e258e5ec-224a-4747-9880-0b09bb19265a" containerName="revoke-keys"
Oct 30 21:54:51.247919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247806    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy"
Oct 30 21:54:51.247919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247819    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e258e5ec-224a-4747-9880-0b09bb19265a" containerName="revoke-keys"
Oct 30 21:54:51.247919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247829    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f0d8de-d487-4c5c-9222-f4bac0b23c49" containerName="revoke-keys"
Oct 30 21:54:51.247919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.247849    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c41b0e-4cdb-47e3-aa79-c5b22a123c62" containerName="istio-proxy"
Oct 30 21:54:51.252988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.252966    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:51.262509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.262481    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"]
Oct 30 21:54:51.309668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.309639    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:51.309858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.309742    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:54:51.353994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.353957    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrg9s\" (UniqueName: \"kubernetes.io/projected/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-kube-api-access-rrg9s\") pod \"maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf\" (UID: \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:51.354238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.354218    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf\" (UID: \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:51.354345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.354327    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf\" (UID: \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:51.457416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.456302    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf\" (UID: \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:51.457416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.456438    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf\" (UID: \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:51.457416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.456577    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrg9s\" (UniqueName: \"kubernetes.io/projected/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-kube-api-access-rrg9s\") pod \"maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf\" (UID: \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:51.461414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.458372    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf\" (UID: \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:51.466462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.465916    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf\" (UID: \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:51.469427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.469339    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrg9s\" (UniqueName: \"kubernetes.io/projected/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-kube-api-access-rrg9s\") pod \"maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf\" (UID: \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:51.569525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.569447    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:51.857134 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:51.857100    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9f4eba_7649_4594_86a8_b5dbd14e6d57.slice/crio-14c5f246fb6c2e371d14da3bc6eda5fff01679790a9099ba70f66e981cea7bd0 WatchSource:0}: Error finding container 14c5f246fb6c2e371d14da3bc6eda5fff01679790a9099ba70f66e981cea7bd0: Status 404 returned error can't find the container with id 14c5f246fb6c2e371d14da3bc6eda5fff01679790a9099ba70f66e981cea7bd0
Oct 30 21:54:51.857908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:51.857882    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"]
Oct 30 21:54:52.660948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:52.660855    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"]
Oct 30 21:54:52.844790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:52.844756    4918 generic.go:356] "Generic (PLEG): container finished" podID="2f9f4eba-7649-4594-86a8-b5dbd14e6d57" containerID="d8c8b7dde0fcc7fb379ca383980d4fd573fd866ea1b8b473c4fcf15d7e92a666" exitCode=0
Oct 30 21:54:52.844911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:52.844842    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf" event={"ID":"2f9f4eba-7649-4594-86a8-b5dbd14e6d57","Type":"ContainerDied","Data":"d8c8b7dde0fcc7fb379ca383980d4fd573fd866ea1b8b473c4fcf15d7e92a666"}
Oct 30 21:54:52.844911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:52.844894    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf" event={"ID":"2f9f4eba-7649-4594-86a8-b5dbd14e6d57","Type":"ContainerStarted","Data":"14c5f246fb6c2e371d14da3bc6eda5fff01679790a9099ba70f66e981cea7bd0"}
Oct 30 21:54:53.086414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:53.084621    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v" podUID="c1ef6ad5-f0ac-4096-8682-bfcbb76437a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:54:54.036741 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.036716    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:54.113612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.113516    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-kube-api-access-rrg9s\" (UniqueName: \"kubernetes.io/projected/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-kube-api-access-rrg9s\") pod \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\" (UID: \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\") "
Oct 30 21:54:54.113612 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.113574    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-xks-ca\") pod \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\" (UID: \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\") "
Oct 30 21:54:54.113920 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.113668    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-openshift-ca\") pod \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\" (UID: \"2f9f4eba-7649-4594-86a8-b5dbd14e6d57\") "
Oct 30 21:54:54.114462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.114315    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-openshift-ca" pod "2f9f4eba-7649-4594-86a8-b5dbd14e6d57" (UID: "2f9f4eba-7649-4594-86a8-b5dbd14e6d57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:54.116698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.116676    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-xks-ca" pod "2f9f4eba-7649-4594-86a8-b5dbd14e6d57" (UID: "2f9f4eba-7649-4594-86a8-b5dbd14e6d57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:54.116950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.116931    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-kube-api-access-rrg9s" pod "2f9f4eba-7649-4594-86a8-b5dbd14e6d57" (UID: "2f9f4eba-7649-4594-86a8-b5dbd14e6d57"). InnerVolumeSpecName "kube-api-access-rrg9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:54.215896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.215865    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrg9s\" (UniqueName: \"kubernetes.io/projected/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-kube-api-access-rrg9s\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:54.215896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.215892    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:54.216116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.215905    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9f4eba-7649-4594-86a8-b5dbd14e6d57-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:54.863888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.863856    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf" event={"ID":"2f9f4eba-7649-4594-86a8-b5dbd14e6d57","Type":"ContainerDied","Data":"14c5f246fb6c2e371d14da3bc6eda5fff01679790a9099ba70f66e981cea7bd0"}
Oct 30 21:54:54.864068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.863893    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"
Oct 30 21:54:54.864068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:54.863903    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14c5f246fb6c2e371d14da3bc6eda5fff01679790a9099ba70f66e981cea7bd0"
Oct 30 21:54:55.733431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:55.733380    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"]
Oct 30 21:54:55.734222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:55.734189    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy" containerID="cri-o://8b4949621a2fe48fb367cdd8011c4509231cfbf111c108cde3751c5482a8260a" gracePeriod=30
Oct 30 21:54:56.264261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.264231    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"]
Oct 30 21:54:56.264962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.264946    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2f9f4eba-7649-4594-86a8-b5dbd14e6d57" containerName="revoke-keys"
Oct 30 21:54:56.265014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.264964    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9f4eba-7649-4594-86a8-b5dbd14e6d57" containerName="revoke-keys"
Oct 30 21:54:56.265096 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.265087    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9f4eba-7649-4594-86a8-b5dbd14e6d57" containerName="revoke-keys"
Oct 30 21:54:56.265147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.265101    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d7835c-782d-400b-8c1e-3a4654fd8a12" containerName="istio-proxy"
Oct 30 21:54:56.268949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.268935    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:54:56.281975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.281947    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"]
Oct 30 21:54:56.332784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.332745    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:56.332784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.332761    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:54:56.367446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.367412    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8\" (UID: \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:54:56.367627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.367582    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8\" (UID: \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:54:56.367828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.367809    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwklc\" (UniqueName: \"kubernetes.io/projected/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-kube-api-access-rwklc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8\" (UID: \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:54:56.469941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.469907    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwklc\" (UniqueName: \"kubernetes.io/projected/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-kube-api-access-rwklc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8\" (UID: \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:54:56.470112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.469986    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8\" (UID: \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:54:56.470112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.470064    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8\" (UID: \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:54:56.470831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.470770    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8\" (UID: \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:54:56.472624 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.472596    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8\" (UID: \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:54:56.484683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.484657    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwklc\" (UniqueName: \"kubernetes.io/projected/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-kube-api-access-rwklc\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8\" (UID: \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:54:56.587658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.587563    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:54:56.893033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.893003    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"]
Oct 30 21:54:56.894060 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:56.894024    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875c1c43_2a86_4bd5_95dd_dc8f9dc3afd5.slice/crio-bed075164e371740b243abd0851215ee64070e8673e72f5fae5a04c9b2e8466b WatchSource:0}: Error finding container bed075164e371740b243abd0851215ee64070e8673e72f5fae5a04c9b2e8466b: Status 404 returned error can't find the container with id bed075164e371740b243abd0851215ee64070e8673e72f5fae5a04c9b2e8466b
Oct 30 21:54:56.932720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:56.932695    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8" event={"ID":"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5","Type":"ContainerStarted","Data":"bed075164e371740b243abd0851215ee64070e8673e72f5fae5a04c9b2e8466b"}
Oct 30 21:54:57.143851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.143779    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"]
Oct 30 21:54:57.149355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.149338    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:54:57.158956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.158660    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"]
Oct 30 21:54:57.289093 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.289063    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d9a021-cddb9\" (UID: \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:54:57.289267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.289121    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-588hc\" (UniqueName: \"kubernetes.io/projected/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-kube-api-access-588hc\") pod \"maas-api-revoke-sub-a3d9a021-cddb9\" (UID: \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:54:57.289315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.289280    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d9a021-cddb9\" (UID: \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:54:57.390553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.390517    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d9a021-cddb9\" (UID: \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:54:57.390744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.390690    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d9a021-cddb9\" (UID: \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:54:57.390744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.390720    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-588hc\" (UniqueName: \"kubernetes.io/projected/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-kube-api-access-588hc\") pod \"maas-api-revoke-sub-a3d9a021-cddb9\" (UID: \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:54:57.391182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.391157    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d9a021-cddb9\" (UID: \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:54:57.393184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.393160    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d9a021-cddb9\" (UID: \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:54:57.400737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.400688    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-588hc\" (UniqueName: \"kubernetes.io/projected/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-kube-api-access-588hc\") pod \"maas-api-revoke-sub-a3d9a021-cddb9\" (UID: \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:54:57.467160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.467135    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:54:57.661403 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.661304    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g" podUID="806c4d25-beb2-45d7-9df8-a107c39b08dd" containerName="payload-pre-processing" containerID="cri-o://d339d335eba4d0afef0918350d984a827daf01a9ee8f888eb1095b34b6bfc8e1" gracePeriod=25
Oct 30 21:54:57.749918 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:54:57.749884    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1f06a3_7b8e_47cb_b7d4_418250d4f2bf.slice/crio-6a958c960d4865e63388ea8c644b06c197fee5bf228d675aae61b1cb1e9c4207 WatchSource:0}: Error finding container 6a958c960d4865e63388ea8c644b06c197fee5bf228d675aae61b1cb1e9c4207: Status 404 returned error can't find the container with id 6a958c960d4865e63388ea8c644b06c197fee5bf228d675aae61b1cb1e9c4207
Oct 30 21:54:57.750635 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.750610    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"]
Oct 30 21:54:57.948913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.948824    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9" event={"ID":"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf","Type":"ContainerStarted","Data":"6a958c960d4865e63388ea8c644b06c197fee5bf228d675aae61b1cb1e9c4207"}
Oct 30 21:54:57.951345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.951316    4918 generic.go:356] "Generic (PLEG): container finished" podID="806c4d25-beb2-45d7-9df8-a107c39b08dd" containerID="d339d335eba4d0afef0918350d984a827daf01a9ee8f888eb1095b34b6bfc8e1" exitCode=0
Oct 30 21:54:57.951522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.951451    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g" event={"ID":"806c4d25-beb2-45d7-9df8-a107c39b08dd","Type":"ContainerDied","Data":"d339d335eba4d0afef0918350d984a827daf01a9ee8f888eb1095b34b6bfc8e1"}
Oct 30 21:54:57.954538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:57.953714    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8" event={"ID":"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5","Type":"ContainerStarted","Data":"c65ec90234be24e4b8ad3a76b651b4f9beabed3c8be9478574ac4250b7bd349a"}
Oct 30 21:54:58.026061 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.025898    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8" podStartSLOduration=2.025877054 podStartE2EDuration="2.025877054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:57.982536062 +0000 UTC m=+3211.029095230" watchObservedRunningTime="2026-10-30 21:54:58.025877054 +0000 UTC m=+3211.072436201"
Oct 30 21:54:58.099859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.099834    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"
Oct 30 21:54:58.222558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.222519    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806c4d25-beb2-45d7-9df8-a107c39b08dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806c4d25-beb2-45d7-9df8-a107c39b08dd-plugins-config-volume\") pod \"806c4d25-beb2-45d7-9df8-a107c39b08dd\" (UID: \"806c4d25-beb2-45d7-9df8-a107c39b08dd\") "
Oct 30 21:54:58.222757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.222678    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806c4d25-beb2-45d7-9df8-a107c39b08dd-kube-api-access-5x42k\" (UniqueName: \"kubernetes.io/projected/806c4d25-beb2-45d7-9df8-a107c39b08dd-kube-api-access-5x42k\") pod \"806c4d25-beb2-45d7-9df8-a107c39b08dd\" (UID: \"806c4d25-beb2-45d7-9df8-a107c39b08dd\") "
Oct 30 21:54:58.222975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.222946    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806c4d25-beb2-45d7-9df8-a107c39b08dd-plugins-config-volume" pod "806c4d25-beb2-45d7-9df8-a107c39b08dd" (UID: "806c4d25-beb2-45d7-9df8-a107c39b08dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:58.224359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.224330    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/806c4d25-beb2-45d7-9df8-a107c39b08dd-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:58.225637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.225617    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806c4d25-beb2-45d7-9df8-a107c39b08dd-kube-api-access-5x42k" pod "806c4d25-beb2-45d7-9df8-a107c39b08dd" (UID: "806c4d25-beb2-45d7-9df8-a107c39b08dd"). InnerVolumeSpecName "kube-api-access-5x42k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:58.326308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.326279    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x42k\" (UniqueName: \"kubernetes.io/projected/806c4d25-beb2-45d7-9df8-a107c39b08dd-kube-api-access-5x42k\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:58.963261 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.963230    4918 generic.go:356] "Generic (PLEG): container finished" podID="fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf" containerID="ddbb99d0a4f38e2417b9409f95df1726fa14f3ff9d5df8df09bc96e7ac5b4a63" exitCode=0
Oct 30 21:54:58.963726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.963323    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9" event={"ID":"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf","Type":"ContainerDied","Data":"ddbb99d0a4f38e2417b9409f95df1726fa14f3ff9d5df8df09bc96e7ac5b4a63"}
Oct 30 21:54:58.964959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.964939    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"
Oct 30 21:54:58.965105 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.965002    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g" event={"ID":"806c4d25-beb2-45d7-9df8-a107c39b08dd","Type":"ContainerDied","Data":"76db9225162a61bd01d804d3251171f5cccc5592c4bc8e46cf483748f78cbd55"}
Oct 30 21:54:58.965105 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.965064    4918 scope.go:122] "RemoveContainer" containerID="d339d335eba4d0afef0918350d984a827daf01a9ee8f888eb1095b34b6bfc8e1"
Oct 30 21:54:58.967005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.966976    4918 generic.go:356] "Generic (PLEG): container finished" podID="875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5" containerID="c65ec90234be24e4b8ad3a76b651b4f9beabed3c8be9478574ac4250b7bd349a" exitCode=0
Oct 30 21:54:58.967187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:58.967071    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8" event={"ID":"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5","Type":"ContainerDied","Data":"c65ec90234be24e4b8ad3a76b651b4f9beabed3c8be9478574ac4250b7bd349a"}
Oct 30 21:54:59.024943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:59.024911    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"]
Oct 30 21:54:59.030752 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:59.030728    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-15e79c-89f86664c-cjd4g"]
Oct 30 21:54:59.080348 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:54:59.080259    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806c4d25-beb2-45d7-9df8-a107c39b08dd" path="/var/lib/kubelet/pods/806c4d25-beb2-45d7-9df8-a107c39b08dd/volumes"
Oct 30 21:54:59.182412 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:54:59.179708    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:55:00.192789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.192703    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:55:00.195978 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.195958    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:55:00.254637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.254555    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-openshift-ca\") pod \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\" (UID: \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\") "
Oct 30 21:55:00.254637 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.254620    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-xks-ca\") pod \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\" (UID: \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\") "
Oct 30 21:55:00.254879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.254663    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-kube-api-access-588hc\" (UniqueName: \"kubernetes.io/projected/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-kube-api-access-588hc\") pod \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\" (UID: \"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf\") "
Oct 30 21:55:00.254879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.254725    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-xks-ca\") pod \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\" (UID: \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\") "
Oct 30 21:55:00.254879 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.254760    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-kube-api-access-rwklc\" (UniqueName: \"kubernetes.io/projected/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-kube-api-access-rwklc\") pod \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\" (UID: \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\") "
Oct 30 21:55:00.255079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.254941    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-openshift-ca" pod "fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf" (UID: "fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:00.255079 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.255013    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-openshift-ca\") pod \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\" (UID: \"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5\") "
Oct 30 21:55:00.255676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.255646    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:00.255957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.255776    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-openshift-ca" pod "875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5" (UID: "875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:00.258294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.258149    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-xks-ca" pod "875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5" (UID: "875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:00.258294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.258156    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-xks-ca" pod "fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf" (UID: "fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:00.258767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.258699    4918 patch_prober.go:29] interesting pod/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:55:00.258767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.258754    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:55:00.259295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.259256    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-kube-api-access-588hc" pod "fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf" (UID: "fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf"). InnerVolumeSpecName "kube-api-access-588hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:00.259599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.259575    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-kube-api-access-rwklc" pod "875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5" (UID: "875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5"). InnerVolumeSpecName "kube-api-access-rwklc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:00.356998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.356970    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:00.356998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.356996    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-588hc\" (UniqueName: \"kubernetes.io/projected/fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf-kube-api-access-588hc\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:00.357153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.357006    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:00.357153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.357014    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwklc\" (UniqueName: \"kubernetes.io/projected/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-kube-api-access-rwklc\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:00.357153 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.357023    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:00.993758 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.993728    4918 generic.go:356] "Generic (PLEG): container finished" podID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerID="8b4949621a2fe48fb367cdd8011c4509231cfbf111c108cde3751c5482a8260a" exitCode=0
Oct 30 21:55:00.993917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.993822    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" event={"ID":"be5b070d-b6a5-4eb0-be88-300467b249f9","Type":"ContainerDied","Data":"8b4949621a2fe48fb367cdd8011c4509231cfbf111c108cde3751c5482a8260a"}
Oct 30 21:55:00.993917 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.993888    4918 scope.go:122] "RemoveContainer" containerID="51dc0ec76e9d0e56db1226c4800fe338cc898383a972123c56422222bab05250"
Oct 30 21:55:00.995440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.995423    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9"
Oct 30 21:55:00.995559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.995453    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d9a021-cddb9" event={"ID":"fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf","Type":"ContainerDied","Data":"6a958c960d4865e63388ea8c644b06c197fee5bf228d675aae61b1cb1e9c4207"}
Oct 30 21:55:00.995559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.995503    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a958c960d4865e63388ea8c644b06c197fee5bf228d675aae61b1cb1e9c4207"
Oct 30 21:55:00.997281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.997264    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"
Oct 30 21:55:00.997433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.997337    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8" event={"ID":"875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5","Type":"ContainerDied","Data":"bed075164e371740b243abd0851215ee64070e8673e72f5fae5a04c9b2e8466b"}
Oct 30 21:55:00.997515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:00.997381    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bed075164e371740b243abd0851215ee64070e8673e72f5fae5a04c9b2e8466b"
Oct 30 21:55:01.003166 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.003121    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:55:01.069594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.069551    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-envoy\") pod \"be5b070d-b6a5-4eb0-be88-300467b249f9\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") "
Oct 30 21:55:01.070195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.070156    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-kube-api-access-d2jbj\" (UniqueName: \"kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-kube-api-access-d2jbj\") pod \"be5b070d-b6a5-4eb0-be88-300467b249f9\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") "
Oct 30 21:55:01.070338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.070230    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-certs\") pod \"be5b070d-b6a5-4eb0-be88-300467b249f9\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") "
Oct 30 21:55:01.070338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.070289    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-token\" (UniqueName: \"kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-token\") pod \"be5b070d-b6a5-4eb0-be88-300467b249f9\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") "
Oct 30 21:55:01.070338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.070326    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-socket\") pod \"be5b070d-b6a5-4eb0-be88-300467b249f9\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") "
Oct 30 21:55:01.070565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.070403    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be5b070d-b6a5-4eb0-be88-300467b249f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5b070d-b6a5-4eb0-be88-300467b249f9-istiod-ca-cert\") pod \"be5b070d-b6a5-4eb0-be88-300467b249f9\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") "
Oct 30 21:55:01.070628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.070612    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-data\") pod \"be5b070d-b6a5-4eb0-be88-300467b249f9\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") "
Oct 30 21:55:01.070686 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.070652    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-podinfo\") pod \"be5b070d-b6a5-4eb0-be88-300467b249f9\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") "
Oct 30 21:55:01.070778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.070687    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-credential-socket\") pod \"be5b070d-b6a5-4eb0-be88-300467b249f9\" (UID: \"be5b070d-b6a5-4eb0-be88-300467b249f9\") "
Oct 30 21:55:01.070841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.070782    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-socket" pod "be5b070d-b6a5-4eb0-be88-300467b249f9" (UID: "be5b070d-b6a5-4eb0-be88-300467b249f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:01.070999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.070978    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-data" pod "be5b070d-b6a5-4eb0-be88-300467b249f9" (UID: "be5b070d-b6a5-4eb0-be88-300467b249f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:01.070999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.070906    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be5b070d-b6a5-4eb0-be88-300467b249f9-istiod-ca-cert" pod "be5b070d-b6a5-4eb0-be88-300467b249f9" (UID: "be5b070d-b6a5-4eb0-be88-300467b249f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:01.071252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.071228    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-credential-socket" pod "be5b070d-b6a5-4eb0-be88-300467b249f9" (UID: "be5b070d-b6a5-4eb0-be88-300467b249f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:01.071494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.071467    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.071494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.071496    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.071742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.071510    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.071742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.071525    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be5b070d-b6a5-4eb0-be88-300467b249f9-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.071878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.071757    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-certs" pod "be5b070d-b6a5-4eb0-be88-300467b249f9" (UID: "be5b070d-b6a5-4eb0-be88-300467b249f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:01.074208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.074169    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-envoy" pod "be5b070d-b6a5-4eb0-be88-300467b249f9" (UID: "be5b070d-b6a5-4eb0-be88-300467b249f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:01.074626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.074602    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-kube-api-access-d2jbj" pod "be5b070d-b6a5-4eb0-be88-300467b249f9" (UID: "be5b070d-b6a5-4eb0-be88-300467b249f9"). InnerVolumeSpecName "kube-api-access-d2jbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:01.075044 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.075010    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-token" pod "be5b070d-b6a5-4eb0-be88-300467b249f9" (UID: "be5b070d-b6a5-4eb0-be88-300467b249f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:01.076009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.075986    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-podinfo" pod "be5b070d-b6a5-4eb0-be88-300467b249f9" (UID: "be5b070d-b6a5-4eb0-be88-300467b249f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:55:01.172590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.172557    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.172590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.172588    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2jbj\" (UniqueName: \"kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-kube-api-access-d2jbj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.172794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.172598    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be5b070d-b6a5-4eb0-be88-300467b249f9-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.172794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.172610    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.172794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.172621    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be5b070d-b6a5-4eb0-be88-300467b249f9-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:01.364836 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:01.361243    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"]
Oct 30 21:55:02.007753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:02.007723    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"
Oct 30 21:55:02.007962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:02.007751    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv" event={"ID":"be5b070d-b6a5-4eb0-be88-300467b249f9","Type":"ContainerDied","Data":"70962f7c8aa163a8c96f4c60093135816c093303ff597af13c0376aa15b20d63"}
Oct 30 21:55:02.007962 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:02.007820    4918 scope.go:122] "RemoveContainer" containerID="8b4949621a2fe48fb367cdd8011c4509231cfbf111c108cde3751c5482a8260a"
Oct 30 21:55:02.052682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:02.052653    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"]
Oct 30 21:55:02.057194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:02.057174    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-36a099-openshift-default-579dc448d5-wxjrv"]
Oct 30 21:55:03.066653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:03.066617    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v" podUID="c1ef6ad5-f0ac-4096-8682-bfcbb76437a3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:03.078872 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:03.078846    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" path="/var/lib/kubelet/pods/be5b070d-b6a5-4eb0-be88-300467b249f9/volumes"
Oct 30 21:55:04.435969 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:04.435945    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"
Oct 30 21:55:04.557413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:04.547888    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-plugins-config-volume\") pod \"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3\" (UID: \"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3\") "
Oct 30 21:55:04.557413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:04.548197    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-kube-api-access-lckvt\" (UniqueName: \"kubernetes.io/projected/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-kube-api-access-lckvt\") pod \"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3\" (UID: \"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3\") "
Oct 30 21:55:04.580268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:04.575914    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-plugins-config-volume" pod "c1ef6ad5-f0ac-4096-8682-bfcbb76437a3" (UID: "c1ef6ad5-f0ac-4096-8682-bfcbb76437a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:04.580268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:04.576285    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:04.580268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:04.576487    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-kube-api-access-lckvt" pod "c1ef6ad5-f0ac-4096-8682-bfcbb76437a3" (UID: "c1ef6ad5-f0ac-4096-8682-bfcbb76437a3"). InnerVolumeSpecName "kube-api-access-lckvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:04.678118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:04.678002    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lckvt\" (UniqueName: \"kubernetes.io/projected/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3-kube-api-access-lckvt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:04.774368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:04.774326    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"]
Oct 30 21:55:04.779039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:04.779005    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f9e5a7-92dhf"]
Oct 30 21:55:05.051737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:05.051704    4918 generic.go:356] "Generic (PLEG): container finished" podID="c1ef6ad5-f0ac-4096-8682-bfcbb76437a3" containerID="d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969" exitCode=0
Oct 30 21:55:05.051934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:05.051759    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"
Oct 30 21:55:05.051934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:05.051778    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v" event={"ID":"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3","Type":"ContainerDied","Data":"d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969"}
Oct 30 21:55:05.051934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:05.051850    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v" event={"ID":"c1ef6ad5-f0ac-4096-8682-bfcbb76437a3","Type":"ContainerDied","Data":"e5718cd30b764bab7cc1cf3b804119ad9374e1b5fef709b7370d1a96f2571767"}
Oct 30 21:55:05.051934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:05.051876    4918 scope.go:122] "RemoveContainer" containerID="d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969"
Oct 30 21:55:05.064294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:05.064278    4918 scope.go:122] "RemoveContainer" containerID="d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969"
Oct 30 21:55:05.064593 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:05.064567    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969\": container with ID starting with d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969 not found: ID does not exist" containerID="d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969"
Oct 30 21:55:05.064656 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:05.064597    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969"} err="failed to get container status \"d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969\": rpc error: code = NotFound desc = could not find container \"d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969\": container with ID starting with d71cd07fb259591c00771ff0ca6ec3a21755fdcbad72fad2b8100aba97891969 not found: ID does not exist"
Oct 30 21:55:05.079939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:05.079907    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f9f4eba-7649-4594-86a8-b5dbd14e6d57" path="/var/lib/kubelet/pods/2f9f4eba-7649-4594-86a8-b5dbd14e6d57/volumes"
Oct 30 21:55:05.080284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:05.080268    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"]
Oct 30 21:55:05.083055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:05.083032    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-36a099-669fcc5bc5-2t98v"]
Oct 30 21:55:06.362201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:06.362155    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk" podUID="ddfed72d-ef13-4d74-9cae-cd8758f202a0" containerName="payload-pre-processing" containerID="cri-o://5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72" gracePeriod=25
Oct 30 21:55:06.584216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:06.581468    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"]
Oct 30 21:55:06.592041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:06.591696    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"]
Oct 30 21:55:06.676550 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:06.676525    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"
Oct 30 21:55:06.814468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:06.814437    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddfed72d-ef13-4d74-9cae-cd8758f202a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfed72d-ef13-4d74-9cae-cd8758f202a0-plugins-config-volume\") pod \"ddfed72d-ef13-4d74-9cae-cd8758f202a0\" (UID: \"ddfed72d-ef13-4d74-9cae-cd8758f202a0\") "
Oct 30 21:55:06.814652 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:06.814625    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfed72d-ef13-4d74-9cae-cd8758f202a0-kube-api-access-5c4k4\" (UniqueName: \"kubernetes.io/projected/ddfed72d-ef13-4d74-9cae-cd8758f202a0-kube-api-access-5c4k4\") pod \"ddfed72d-ef13-4d74-9cae-cd8758f202a0\" (UID: \"ddfed72d-ef13-4d74-9cae-cd8758f202a0\") "
Oct 30 21:55:06.814812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:06.814790    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfed72d-ef13-4d74-9cae-cd8758f202a0-plugins-config-volume" pod "ddfed72d-ef13-4d74-9cae-cd8758f202a0" (UID: "ddfed72d-ef13-4d74-9cae-cd8758f202a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:06.815159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:06.815140    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfed72d-ef13-4d74-9cae-cd8758f202a0-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:06.816881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:06.816859    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfed72d-ef13-4d74-9cae-cd8758f202a0-kube-api-access-5c4k4" pod "ddfed72d-ef13-4d74-9cae-cd8758f202a0" (UID: "ddfed72d-ef13-4d74-9cae-cd8758f202a0"). InnerVolumeSpecName "kube-api-access-5c4k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:06.916345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:06.916272    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c4k4\" (UniqueName: \"kubernetes.io/projected/ddfed72d-ef13-4d74-9cae-cd8758f202a0-kube-api-access-5c4k4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:07.136287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:07.136244    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ef6ad5-f0ac-4096-8682-bfcbb76437a3" path="/var/lib/kubelet/pods/c1ef6ad5-f0ac-4096-8682-bfcbb76437a3/volumes"
Oct 30 21:55:07.136870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:07.136842    4918 generic.go:356] "Generic (PLEG): container finished" podID="ddfed72d-ef13-4d74-9cae-cd8758f202a0" containerID="5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72" exitCode=0
Oct 30 21:55:07.136870 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:07.136861    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk" event={"ID":"ddfed72d-ef13-4d74-9cae-cd8758f202a0","Type":"ContainerDied","Data":"5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72"}
Oct 30 21:55:07.137005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:07.136904    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk" event={"ID":"ddfed72d-ef13-4d74-9cae-cd8758f202a0","Type":"ContainerDied","Data":"4e6b9538ebfa9769d32e63bde5705d7f45b3429fdebdc43754d28f2c36809852"}
Oct 30 21:55:07.137005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:07.136927    4918 scope.go:122] "RemoveContainer" containerID="5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72"
Oct 30 21:55:07.137005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:07.136958    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"
Oct 30 21:55:07.152107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:07.152083    4918 scope.go:122] "RemoveContainer" containerID="5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72"
Oct 30 21:55:07.152464 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:07.152419    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72\": container with ID starting with 5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72 not found: ID does not exist" containerID="5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72"
Oct 30 21:55:07.152464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:07.152458    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72"} err="failed to get container status \"5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72\": rpc error: code = NotFound desc = could not find container \"5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72\": container with ID starting with 5bbb21d4e9f75da316206f373ac9014bb245b2f7979a127ee67c52553076ef72 not found: ID does not exist"
Oct 30 21:55:07.172563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:07.172489    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"]
Oct 30 21:55:07.185671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:07.185643    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f9e5a7-5d4dcbd95-8fggk"]
Oct 30 21:55:08.133910 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.133878    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"]
Oct 30 21:55:08.134557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134542    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ddfed72d-ef13-4d74-9cae-cd8758f202a0" containerName="payload-pre-processing"
Oct 30 21:55:08.134557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134557    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddfed72d-ef13-4d74-9cae-cd8758f202a0" containerName="payload-pre-processing"
Oct 30 21:55:08.134648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134574    4918 cpu_manager.go:414] "containerMap: removing container" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy"
Oct 30 21:55:08.134648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134580    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy"
Oct 30 21:55:08.134648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134593    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf" containerName="revoke-keys"
Oct 30 21:55:08.134648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134597    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf" containerName="revoke-keys"
Oct 30 21:55:08.134648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134607    4918 cpu_manager.go:414] "containerMap: removing container" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy"
Oct 30 21:55:08.134648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134612    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy"
Oct 30 21:55:08.134648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134634    4918 cpu_manager.go:414] "containerMap: removing container" podUID="806c4d25-beb2-45d7-9df8-a107c39b08dd" containerName="payload-pre-processing"
Oct 30 21:55:08.134648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134639    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806c4d25-beb2-45d7-9df8-a107c39b08dd" containerName="payload-pre-processing"
Oct 30 21:55:08.134648 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134650    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c1ef6ad5-f0ac-4096-8682-bfcbb76437a3" containerName="payload-pre-processing"
Oct 30 21:55:08.134936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134655    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ef6ad5-f0ac-4096-8682-bfcbb76437a3" containerName="payload-pre-processing"
Oct 30 21:55:08.134936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134662    4918 cpu_manager.go:414] "containerMap: removing container" podUID="875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5" containerName="revoke-keys"
Oct 30 21:55:08.134936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134667    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5" containerName="revoke-keys"
Oct 30 21:55:08.134936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134756    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf" containerName="revoke-keys"
Oct 30 21:55:08.134936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134765    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy"
Oct 30 21:55:08.134936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134774    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="806c4d25-beb2-45d7-9df8-a107c39b08dd" containerName="payload-pre-processing"
Oct 30 21:55:08.134936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134779    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ef6ad5-f0ac-4096-8682-bfcbb76437a3" containerName="payload-pre-processing"
Oct 30 21:55:08.134936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134785    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5" containerName="revoke-keys"
Oct 30 21:55:08.134936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134793    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddfed72d-ef13-4d74-9cae-cd8758f202a0" containerName="payload-pre-processing"
Oct 30 21:55:08.134936 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.134800    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="be5b070d-b6a5-4eb0-be88-300467b249f9" containerName="istio-proxy"
Oct 30 21:55:08.139731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.139704    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:08.152074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.152047    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:55:08.152074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.152063    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:55:08.152275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.152224    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"]
Oct 30 21:55:08.164914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.164893    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-adf308-8959x\" (UID: \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:08.165075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.164950    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-adf308-8959x\" (UID: \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:08.165210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.165185    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx9t9\" (UniqueName: \"kubernetes.io/projected/28b82edd-4669-4e18-87b8-9aba34e2ca1d-kube-api-access-tx9t9\") pod \"maas-api-revoke-keys-e2e-inf-a-adf308-8959x\" (UID: \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:08.266268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.266236    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx9t9\" (UniqueName: \"kubernetes.io/projected/28b82edd-4669-4e18-87b8-9aba34e2ca1d-kube-api-access-tx9t9\") pod \"maas-api-revoke-keys-e2e-inf-a-adf308-8959x\" (UID: \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:08.266464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.266367    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-adf308-8959x\" (UID: \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:08.266561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.266478    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-adf308-8959x\" (UID: \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:08.267055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.267031    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-adf308-8959x\" (UID: \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:08.269005 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.268983    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-adf308-8959x\" (UID: \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:08.278377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.278354    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx9t9\" (UniqueName: \"kubernetes.io/projected/28b82edd-4669-4e18-87b8-9aba34e2ca1d-kube-api-access-tx9t9\") pod \"maas-api-revoke-keys-e2e-inf-a-adf308-8959x\" (UID: \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:08.457206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.456563    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:08.678439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.678404    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"]
Oct 30 21:55:08.690749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.690720    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15e79c-vgqv7"]
Oct 30 21:55:08.770901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:08.770712    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"]
Oct 30 21:55:08.778811 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:55:08.778770    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b82edd_4669_4e18_87b8_9aba34e2ca1d.slice/crio-562191fff3a2123d8b640d11eea70ee893a0de77e89e8793e6e97abb446f9297 WatchSource:0}: Error finding container 562191fff3a2123d8b640d11eea70ee893a0de77e89e8793e6e97abb446f9297: Status 404 returned error can't find the container with id 562191fff3a2123d8b640d11eea70ee893a0de77e89e8793e6e97abb446f9297
Oct 30 21:55:09.081399 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:09.081310    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f0d8de-d487-4c5c-9222-f4bac0b23c49" path="/var/lib/kubelet/pods/05f0d8de-d487-4c5c-9222-f4bac0b23c49/volumes"
Oct 30 21:55:09.081839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:09.081821    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddfed72d-ef13-4d74-9cae-cd8758f202a0" path="/var/lib/kubelet/pods/ddfed72d-ef13-4d74-9cae-cd8758f202a0/volumes"
Oct 30 21:55:09.156728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:09.156689    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x" event={"ID":"28b82edd-4669-4e18-87b8-9aba34e2ca1d","Type":"ContainerStarted","Data":"562191fff3a2123d8b640d11eea70ee893a0de77e89e8793e6e97abb446f9297"}
Oct 30 21:55:10.166619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:10.166581    4918 generic.go:356] "Generic (PLEG): container finished" podID="28b82edd-4669-4e18-87b8-9aba34e2ca1d" containerID="fb73e07543d40828d0188a2ce2298388810ee50cff105a70403ad23590e10dbd" exitCode=0
Oct 30 21:55:10.167119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:10.166616    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x" event={"ID":"28b82edd-4669-4e18-87b8-9aba34e2ca1d","Type":"ContainerDied","Data":"fb73e07543d40828d0188a2ce2298388810ee50cff105a70403ad23590e10dbd"}
Oct 30 21:55:11.405912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:11.405887    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:11.520968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:11.520921    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-xks-ca\") pod \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\" (UID: \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\") "
Oct 30 21:55:11.521170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:11.521158    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-openshift-ca\") pod \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\" (UID: \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\") "
Oct 30 21:55:11.521238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:11.521209    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b82edd-4669-4e18-87b8-9aba34e2ca1d-kube-api-access-tx9t9\" (UniqueName: \"kubernetes.io/projected/28b82edd-4669-4e18-87b8-9aba34e2ca1d-kube-api-access-tx9t9\") pod \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\" (UID: \"28b82edd-4669-4e18-87b8-9aba34e2ca1d\") "
Oct 30 21:55:11.521625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:11.521596    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-openshift-ca" pod "28b82edd-4669-4e18-87b8-9aba34e2ca1d" (UID: "28b82edd-4669-4e18-87b8-9aba34e2ca1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:11.523185 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:11.523165    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-xks-ca" pod "28b82edd-4669-4e18-87b8-9aba34e2ca1d" (UID: "28b82edd-4669-4e18-87b8-9aba34e2ca1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:11.523742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:11.523722    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b82edd-4669-4e18-87b8-9aba34e2ca1d-kube-api-access-tx9t9" pod "28b82edd-4669-4e18-87b8-9aba34e2ca1d" (UID: "28b82edd-4669-4e18-87b8-9aba34e2ca1d"). InnerVolumeSpecName "kube-api-access-tx9t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:11.589209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:11.587958    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9" podUID="4515aee4-3a67-43d0-9d60-7ad6524bf8d3" containerName="payload-processing" containerID="cri-o://e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d" gracePeriod=25
Oct 30 21:55:11.605592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:11.605479    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5" podUID="22a44e09-93c1-4e78-bc3e-52de37538ec3" containerName="payload-pre-processing" containerID="cri-o://526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9" gracePeriod=25
Oct 30 21:55:11.645434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:11.644627    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:11.645434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:11.644665    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b82edd-4669-4e18-87b8-9aba34e2ca1d-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:11.645434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:11.644680    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx9t9\" (UniqueName: \"kubernetes.io/projected/28b82edd-4669-4e18-87b8-9aba34e2ca1d-kube-api-access-tx9t9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:12.196794 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:12.196767    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"
Oct 30 21:55:12.196954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:12.196767    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x" event={"ID":"28b82edd-4669-4e18-87b8-9aba34e2ca1d","Type":"ContainerDied","Data":"562191fff3a2123d8b640d11eea70ee893a0de77e89e8793e6e97abb446f9297"}
Oct 30 21:55:12.196954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:12.196900    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="562191fff3a2123d8b640d11eea70ee893a0de77e89e8793e6e97abb446f9297"
Oct 30 21:55:14.994725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:14.994689    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"]
Oct 30 21:55:14.995180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:14.994937    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" podUID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" containerName="istio-proxy" containerID="cri-o://c0bd31ec03c9205b183fee8e9c2cb4d89534902413ffd1adae2b92ce6bb56e8c" gracePeriod=30
Oct 30 21:55:15.622308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.622272    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"]
Oct 30 21:55:15.622961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.622946    4918 cpu_manager.go:414] "containerMap: removing container" podUID="28b82edd-4669-4e18-87b8-9aba34e2ca1d" containerName="revoke-keys"
Oct 30 21:55:15.623034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.622962    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b82edd-4669-4e18-87b8-9aba34e2ca1d" containerName="revoke-keys"
Oct 30 21:55:15.623090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.623080    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b82edd-4669-4e18-87b8-9aba34e2ca1d" containerName="revoke-keys"
Oct 30 21:55:15.627098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.627080    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:15.638535 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.638509    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:55:15.638662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.638646    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:55:15.643021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.642996    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"]
Oct 30 21:55:15.728454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.728425    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs\" (UID: \"208a3de5-1523-4585-8569-f8314c1d2810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:15.728605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.728478    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v756\" (UniqueName: \"kubernetes.io/projected/208a3de5-1523-4585-8569-f8314c1d2810-kube-api-access-6v756\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs\" (UID: \"208a3de5-1523-4585-8569-f8314c1d2810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:15.728605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.728567    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs\" (UID: \"208a3de5-1523-4585-8569-f8314c1d2810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:15.829602 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.829560    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v756\" (UniqueName: \"kubernetes.io/projected/208a3de5-1523-4585-8569-f8314c1d2810-kube-api-access-6v756\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs\" (UID: \"208a3de5-1523-4585-8569-f8314c1d2810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:15.829847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.829626    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs\" (UID: \"208a3de5-1523-4585-8569-f8314c1d2810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:15.829965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.829943    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs\" (UID: \"208a3de5-1523-4585-8569-f8314c1d2810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:15.830630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.830603    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs\" (UID: \"208a3de5-1523-4585-8569-f8314c1d2810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:15.832302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.832275    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs\" (UID: \"208a3de5-1523-4585-8569-f8314c1d2810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:15.839256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.839232    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v756\" (UniqueName: \"kubernetes.io/projected/208a3de5-1523-4585-8569-f8314c1d2810-kube-api-access-6v756\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs\" (UID: \"208a3de5-1523-4585-8569-f8314c1d2810\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:15.942478 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:15.942374    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:16.237583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:16.237555    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"]
Oct 30 21:55:16.240428 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:55:16.240379    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208a3de5_1523_4585_8569_f8314c1d2810.slice/crio-619c9a35af371bd6cfc84fd416ec933a8bdfd3a8f9ab9bbdf345dc4e671b90fb WatchSource:0}: Error finding container 619c9a35af371bd6cfc84fd416ec933a8bdfd3a8f9ab9bbdf345dc4e671b90fb: Status 404 returned error can't find the container with id 619c9a35af371bd6cfc84fd416ec933a8bdfd3a8f9ab9bbdf345dc4e671b90fb
Oct 30 21:55:17.256728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:17.256694    4918 generic.go:356] "Generic (PLEG): container finished" podID="208a3de5-1523-4585-8569-f8314c1d2810" containerID="96300630f973c1829bf924796fa42d2385fefa530260526725abdd3c126daff2" exitCode=0
Oct 30 21:55:17.257131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:17.256776    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs" event={"ID":"208a3de5-1523-4585-8569-f8314c1d2810","Type":"ContainerDied","Data":"96300630f973c1829bf924796fa42d2385fefa530260526725abdd3c126daff2"}
Oct 30 21:55:17.257131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:17.256851    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs" event={"ID":"208a3de5-1523-4585-8569-f8314c1d2810","Type":"ContainerStarted","Data":"619c9a35af371bd6cfc84fd416ec933a8bdfd3a8f9ab9bbdf345dc4e671b90fb"}
Oct 30 21:55:18.296115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.296083    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"]
Oct 30 21:55:18.296539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.296288    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6" podUID="debec1c0-11e0-464c-a798-4cb6f334eb32" containerName="istio-proxy" containerID="cri-o://d667bfc9190a02448fa011b20c8cdd7e61c86a3b3051e6bdc916969767794d1a" gracePeriod=30
Oct 30 21:55:18.397424 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.395526    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"]
Oct 30 21:55:18.513378 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.513328    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"]
Oct 30 21:55:18.607026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.607000    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:18.698993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.698960    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208a3de5-1523-4585-8569-f8314c1d2810-kube-api-access-6v756\" (UniqueName: \"kubernetes.io/projected/208a3de5-1523-4585-8569-f8314c1d2810-kube-api-access-6v756\") pod \"208a3de5-1523-4585-8569-f8314c1d2810\" (UID: \"208a3de5-1523-4585-8569-f8314c1d2810\") "
Oct 30 21:55:18.699167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.699031    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-openshift-ca\") pod \"208a3de5-1523-4585-8569-f8314c1d2810\" (UID: \"208a3de5-1523-4585-8569-f8314c1d2810\") "
Oct 30 21:55:18.699167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.699120    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-xks-ca\") pod \"208a3de5-1523-4585-8569-f8314c1d2810\" (UID: \"208a3de5-1523-4585-8569-f8314c1d2810\") "
Oct 30 21:55:18.699524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.699492    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-openshift-ca" pod "208a3de5-1523-4585-8569-f8314c1d2810" (UID: "208a3de5-1523-4585-8569-f8314c1d2810"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:18.699902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.699882    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:18.701305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.701278    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-xks-ca" pod "208a3de5-1523-4585-8569-f8314c1d2810" (UID: "208a3de5-1523-4585-8569-f8314c1d2810"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:18.701492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.701470    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208a3de5-1523-4585-8569-f8314c1d2810-kube-api-access-6v756" pod "208a3de5-1523-4585-8569-f8314c1d2810" (UID: "208a3de5-1523-4585-8569-f8314c1d2810"). InnerVolumeSpecName "kube-api-access-6v756". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:18.814773 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.814674    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v756\" (UniqueName: \"kubernetes.io/projected/208a3de5-1523-4585-8569-f8314c1d2810-kube-api-access-6v756\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:18.820319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.820291    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/208a3de5-1523-4585-8569-f8314c1d2810-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:18.877462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.877421    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"]
Oct 30 21:55:18.888302 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.888269    4918 cpu_manager.go:414] "containerMap: removing container" podUID="208a3de5-1523-4585-8569-f8314c1d2810" containerName="revoke-keys"
Oct 30 21:55:18.888582 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.888560    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208a3de5-1523-4585-8569-f8314c1d2810" containerName="revoke-keys"
Oct 30 21:55:18.888977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.888946    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="208a3de5-1523-4585-8569-f8314c1d2810" containerName="revoke-keys"
Oct 30 21:55:18.899118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.899091    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"]
Oct 30 21:55:18.899512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:18.899492    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:19.031031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.030998    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de823b54-4177-44d2-b352-c98d6afde970-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr\" (UID: \"de823b54-4177-44d2-b352-c98d6afde970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:19.031210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.031055    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r58vs\" (UniqueName: \"kubernetes.io/projected/de823b54-4177-44d2-b352-c98d6afde970-kube-api-access-r58vs\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr\" (UID: \"de823b54-4177-44d2-b352-c98d6afde970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:19.031210 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.031161    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de823b54-4177-44d2-b352-c98d6afde970-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr\" (UID: \"de823b54-4177-44d2-b352-c98d6afde970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:19.132857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.132821    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de823b54-4177-44d2-b352-c98d6afde970-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr\" (UID: \"de823b54-4177-44d2-b352-c98d6afde970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:19.133069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.132893    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r58vs\" (UniqueName: \"kubernetes.io/projected/de823b54-4177-44d2-b352-c98d6afde970-kube-api-access-r58vs\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr\" (UID: \"de823b54-4177-44d2-b352-c98d6afde970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:19.133069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.132930    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de823b54-4177-44d2-b352-c98d6afde970-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr\" (UID: \"de823b54-4177-44d2-b352-c98d6afde970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:19.133634 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.133611    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de823b54-4177-44d2-b352-c98d6afde970-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr\" (UID: \"de823b54-4177-44d2-b352-c98d6afde970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:19.135399 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.135375    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de823b54-4177-44d2-b352-c98d6afde970-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr\" (UID: \"de823b54-4177-44d2-b352-c98d6afde970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:19.144177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.144149    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r58vs\" (UniqueName: \"kubernetes.io/projected/de823b54-4177-44d2-b352-c98d6afde970-kube-api-access-r58vs\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr\" (UID: \"de823b54-4177-44d2-b352-c98d6afde970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:19.255347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.255316    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:19.281766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.281738    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"
Oct 30 21:55:19.281949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.281767    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs" event={"ID":"208a3de5-1523-4585-8569-f8314c1d2810","Type":"ContainerDied","Data":"619c9a35af371bd6cfc84fd416ec933a8bdfd3a8f9ab9bbdf345dc4e671b90fb"}
Oct 30 21:55:19.281949 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.281837    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="619c9a35af371bd6cfc84fd416ec933a8bdfd3a8f9ab9bbdf345dc4e671b90fb"
Oct 30 21:55:19.544814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:19.544787    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"]
Oct 30 21:55:19.546318 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:55:19.546291    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde823b54_4177_44d2_b352_c98d6afde970.slice/crio-d04b5145a0fd00af413ce14e8fab9645392ca873b50c848c81ed027427f029c4 WatchSource:0}: Error finding container d04b5145a0fd00af413ce14e8fab9645392ca873b50c848c81ed027427f029c4: Status 404 returned error can't find the container with id d04b5145a0fd00af413ce14e8fab9645392ca873b50c848c81ed027427f029c4
Oct 30 21:55:20.294403 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.294348    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr" event={"ID":"de823b54-4177-44d2-b352-c98d6afde970","Type":"ContainerStarted","Data":"5fd5e3d52662151c7c1c47b126ab3d132f27e3e73462f2f3df2a7ee7d751320f"}
Oct 30 21:55:20.294590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.294456    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr" event={"ID":"de823b54-4177-44d2-b352-c98d6afde970","Type":"ContainerStarted","Data":"d04b5145a0fd00af413ce14e8fab9645392ca873b50c848c81ed027427f029c4"}
Oct 30 21:55:20.296041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.296015    4918 generic.go:356] "Generic (PLEG): container finished" podID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" containerID="c0bd31ec03c9205b183fee8e9c2cb4d89534902413ffd1adae2b92ce6bb56e8c" exitCode=0
Oct 30 21:55:20.296173 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.296088    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" event={"ID":"4ddd4fc4-2a3a-450b-898b-81d92045fc65","Type":"ContainerDied","Data":"c0bd31ec03c9205b183fee8e9c2cb4d89534902413ffd1adae2b92ce6bb56e8c"}
Oct 30 21:55:20.318507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.318461    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr" podStartSLOduration=2.318448036 podStartE2EDuration="2.318448036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:20.315373015 +0000 UTC m=+3233.361932178" watchObservedRunningTime="2026-10-30 21:55:20.318448036 +0000 UTC m=+3233.365007175"
Oct 30 21:55:20.408709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.408679    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:55:20.457861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.457825    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-data\") pod \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") "
Oct 30 21:55:20.458039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.457929    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-certs\") pod \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") "
Oct 30 21:55:20.458124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.458064    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-credential-socket\") pod \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") "
Oct 30 21:55:20.458124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.458096    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-token\" (UniqueName: \"kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-token\") pod \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") "
Oct 30 21:55:20.458124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.458110    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-data" pod "4ddd4fc4-2a3a-450b-898b-81d92045fc65" (UID: "4ddd4fc4-2a3a-450b-898b-81d92045fc65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:20.458311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.458131    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istiod-ca-cert\") pod \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") "
Oct 30 21:55:20.458311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.458190    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-envoy\") pod \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") "
Oct 30 21:55:20.458311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.458237    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-certs" pod "4ddd4fc4-2a3a-450b-898b-81d92045fc65" (UID: "4ddd4fc4-2a3a-450b-898b-81d92045fc65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:20.458311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.458262    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-socket\") pod \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") "
Oct 30 21:55:20.458311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.458301    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-podinfo\") pod \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") "
Oct 30 21:55:20.458626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.458351    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-kube-api-access-zbzsc\" (UniqueName: \"kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-kube-api-access-zbzsc\") pod \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\" (UID: \"4ddd4fc4-2a3a-450b-898b-81d92045fc65\") "
Oct 30 21:55:20.458626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.458565    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istiod-ca-cert" pod "4ddd4fc4-2a3a-450b-898b-81d92045fc65" (UID: "4ddd4fc4-2a3a-450b-898b-81d92045fc65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:20.458837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.458810    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-credential-socket" pod "4ddd4fc4-2a3a-450b-898b-81d92045fc65" (UID: "4ddd4fc4-2a3a-450b-898b-81d92045fc65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:20.458837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.458817    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-socket" pod "4ddd4fc4-2a3a-450b-898b-81d92045fc65" (UID: "4ddd4fc4-2a3a-450b-898b-81d92045fc65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:20.459313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.459289    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.459441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.459322    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.459441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.459338    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.459441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.459354    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.459441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.459367    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.460835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.460807    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-token" pod "4ddd4fc4-2a3a-450b-898b-81d92045fc65" (UID: "4ddd4fc4-2a3a-450b-898b-81d92045fc65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:20.461357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.461335    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-envoy" pod "4ddd4fc4-2a3a-450b-898b-81d92045fc65" (UID: "4ddd4fc4-2a3a-450b-898b-81d92045fc65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:20.461464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.461443    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-podinfo" pod "4ddd4fc4-2a3a-450b-898b-81d92045fc65" (UID: "4ddd4fc4-2a3a-450b-898b-81d92045fc65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:55:20.461595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.461577    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-kube-api-access-zbzsc" pod "4ddd4fc4-2a3a-450b-898b-81d92045fc65" (UID: "4ddd4fc4-2a3a-450b-898b-81d92045fc65"). InnerVolumeSpecName "kube-api-access-zbzsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:20.560912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.560878    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.560912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.560910    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.561270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.560926    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ddd4fc4-2a3a-450b-898b-81d92045fc65-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:20.561270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:20.560938    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbzsc\" (UniqueName: \"kubernetes.io/projected/4ddd4fc4-2a3a-450b-898b-81d92045fc65-kube-api-access-zbzsc\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:21.305800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:21.305757    4918 generic.go:356] "Generic (PLEG): container finished" podID="de823b54-4177-44d2-b352-c98d6afde970" containerID="5fd5e3d52662151c7c1c47b126ab3d132f27e3e73462f2f3df2a7ee7d751320f" exitCode=0
Oct 30 21:55:21.306013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:21.305840    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr" event={"ID":"de823b54-4177-44d2-b352-c98d6afde970","Type":"ContainerDied","Data":"5fd5e3d52662151c7c1c47b126ab3d132f27e3e73462f2f3df2a7ee7d751320f"}
Oct 30 21:55:21.307301 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:21.307282    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"
Oct 30 21:55:21.307301 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:21.307291    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp" event={"ID":"4ddd4fc4-2a3a-450b-898b-81d92045fc65","Type":"ContainerDied","Data":"a701267df416a01ccbfd74a8fb0d344a492150065cb0e9a6c05c35edb7a9ea02"}
Oct 30 21:55:21.307547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:21.307336    4918 scope.go:122] "RemoveContainer" containerID="c0bd31ec03c9205b183fee8e9c2cb4d89534902413ffd1adae2b92ce6bb56e8c"
Oct 30 21:55:21.348238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:21.348167    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"]
Oct 30 21:55:21.356269 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:21.356244    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f9e5a7-openshift-default-5495669c7c-24ztp"]
Oct 30 21:55:21.479034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:21.478991    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9" podUID="4515aee4-3a67-43d0-9d60-7ad6524bf8d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:21.479208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:21.479079    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5" podUID="22a44e09-93c1-4e78-bc3e-52de37538ec3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:22.656553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:22.656529    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:22.709509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:22.709427    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de823b54-4177-44d2-b352-c98d6afde970-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de823b54-4177-44d2-b352-c98d6afde970-maas-api-openshift-ca\") pod \"de823b54-4177-44d2-b352-c98d6afde970\" (UID: \"de823b54-4177-44d2-b352-c98d6afde970\") "
Oct 30 21:55:22.709666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:22.709584    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de823b54-4177-44d2-b352-c98d6afde970-kube-api-access-r58vs\" (UniqueName: \"kubernetes.io/projected/de823b54-4177-44d2-b352-c98d6afde970-kube-api-access-r58vs\") pod \"de823b54-4177-44d2-b352-c98d6afde970\" (UID: \"de823b54-4177-44d2-b352-c98d6afde970\") "
Oct 30 21:55:22.709820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:22.709790    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de823b54-4177-44d2-b352-c98d6afde970-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de823b54-4177-44d2-b352-c98d6afde970-maas-api-xks-ca\") pod \"de823b54-4177-44d2-b352-c98d6afde970\" (UID: \"de823b54-4177-44d2-b352-c98d6afde970\") "
Oct 30 21:55:22.709904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:22.709859    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de823b54-4177-44d2-b352-c98d6afde970-maas-api-openshift-ca" pod "de823b54-4177-44d2-b352-c98d6afde970" (UID: "de823b54-4177-44d2-b352-c98d6afde970"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:22.710411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:22.710371    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de823b54-4177-44d2-b352-c98d6afde970-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:22.712555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:22.712530    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de823b54-4177-44d2-b352-c98d6afde970-maas-api-xks-ca" pod "de823b54-4177-44d2-b352-c98d6afde970" (UID: "de823b54-4177-44d2-b352-c98d6afde970"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:22.712659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:22.712606    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de823b54-4177-44d2-b352-c98d6afde970-kube-api-access-r58vs" pod "de823b54-4177-44d2-b352-c98d6afde970" (UID: "de823b54-4177-44d2-b352-c98d6afde970"). InnerVolumeSpecName "kube-api-access-r58vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:22.824030 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:22.814621    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r58vs\" (UniqueName: \"kubernetes.io/projected/de823b54-4177-44d2-b352-c98d6afde970-kube-api-access-r58vs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:22.824030 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:22.814664    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de823b54-4177-44d2-b352-c98d6afde970-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:22.995991 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:22.995944    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"]
Oct 30 21:55:23.000636 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.000608    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-726a18-55qn8"]
Oct 30 21:55:23.078241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.078206    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" path="/var/lib/kubelet/pods/4ddd4fc4-2a3a-450b-898b-81d92045fc65/volumes"
Oct 30 21:55:23.078762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.078747    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5" path="/var/lib/kubelet/pods/875c1c43-2a86-4bd5-95dd-dc8f9dc3afd5/volumes"
Oct 30 21:55:23.412501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.412465    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc" podUID="4db3211f-c998-428e-9364-88befce1cf71" containerName="payload-processing" containerID="cri-o://bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b" gracePeriod=25
Oct 30 21:55:23.427138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.427116    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"
Oct 30 21:55:23.427291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.427153    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr" event={"ID":"de823b54-4177-44d2-b352-c98d6afde970","Type":"ContainerDied","Data":"d04b5145a0fd00af413ce14e8fab9645392ca873b50c848c81ed027427f029c4"}
Oct 30 21:55:23.427291 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.427208    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d04b5145a0fd00af413ce14e8fab9645392ca873b50c848c81ed027427f029c4"
Oct 30 21:55:23.429182 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.429118    4918 generic.go:356] "Generic (PLEG): container finished" podID="debec1c0-11e0-464c-a798-4cb6f334eb32" containerID="d667bfc9190a02448fa011b20c8cdd7e61c86a3b3051e6bdc916969767794d1a" exitCode=0
Oct 30 21:55:23.429359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.429215    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6" event={"ID":"debec1c0-11e0-464c-a798-4cb6f334eb32","Type":"ContainerDied","Data":"d667bfc9190a02448fa011b20c8cdd7e61c86a3b3051e6bdc916969767794d1a"}
Oct 30 21:55:23.514878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.514260    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f" podUID="755a5f2c-b99e-4c7b-893a-d346b9f92d81" containerName="payload-pre-processing" containerID="cri-o://7fce3c830e08331123298305c4bec06d9de0b27b04eee3d3f16a7385d31b0c47" gracePeriod=25
Oct 30 21:55:23.550525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.550491    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:55:23.631158 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.631136    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-socket\") pod \"debec1c0-11e0-464c-a798-4cb6f334eb32\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") "
Oct 30 21:55:23.631236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.631187    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-envoy\") pod \"debec1c0-11e0-464c-a798-4cb6f334eb32\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") "
Oct 30 21:55:23.631281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.631244    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-token\" (UniqueName: \"kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-token\") pod \"debec1c0-11e0-464c-a798-4cb6f334eb32\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") "
Oct 30 21:55:23.631452 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.631419    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/debec1c0-11e0-464c-a798-4cb6f334eb32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debec1c0-11e0-464c-a798-4cb6f334eb32-istiod-ca-cert\") pod \"debec1c0-11e0-464c-a798-4cb6f334eb32\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") "
Oct 30 21:55:23.631578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.631497    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-data\") pod \"debec1c0-11e0-464c-a798-4cb6f334eb32\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") "
Oct 30 21:55:23.631578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.631530    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-kube-api-access-9954z\" (UniqueName: \"kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-kube-api-access-9954z\") pod \"debec1c0-11e0-464c-a798-4cb6f334eb32\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") "
Oct 30 21:55:23.631737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.631606    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-credential-socket\") pod \"debec1c0-11e0-464c-a798-4cb6f334eb32\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") "
Oct 30 21:55:23.631737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.631724    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-podinfo\") pod \"debec1c0-11e0-464c-a798-4cb6f334eb32\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") "
Oct 30 21:55:23.631854 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.631774    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-certs\") pod \"debec1c0-11e0-464c-a798-4cb6f334eb32\" (UID: \"debec1c0-11e0-464c-a798-4cb6f334eb32\") "
Oct 30 21:55:23.632288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.632260    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-socket" pod "debec1c0-11e0-464c-a798-4cb6f334eb32" (UID: "debec1c0-11e0-464c-a798-4cb6f334eb32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.632414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.632297    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/debec1c0-11e0-464c-a798-4cb6f334eb32-istiod-ca-cert" pod "debec1c0-11e0-464c-a798-4cb6f334eb32" (UID: "debec1c0-11e0-464c-a798-4cb6f334eb32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:23.632596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.632543    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debec1c0-11e0-464c-a798-4cb6f334eb32-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.632596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.632570    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.632799 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.632673    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-certs" pod "debec1c0-11e0-464c-a798-4cb6f334eb32" (UID: "debec1c0-11e0-464c-a798-4cb6f334eb32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.633222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.633196    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-credential-socket" pod "debec1c0-11e0-464c-a798-4cb6f334eb32" (UID: "debec1c0-11e0-464c-a798-4cb6f334eb32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.633361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.633340    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-data" pod "debec1c0-11e0-464c-a798-4cb6f334eb32" (UID: "debec1c0-11e0-464c-a798-4cb6f334eb32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.634342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.634319    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-envoy" pod "debec1c0-11e0-464c-a798-4cb6f334eb32" (UID: "debec1c0-11e0-464c-a798-4cb6f334eb32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:23.634784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.634766    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-podinfo" pod "debec1c0-11e0-464c-a798-4cb6f334eb32" (UID: "debec1c0-11e0-464c-a798-4cb6f334eb32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:55:23.635089 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.635069    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-kube-api-access-9954z" pod "debec1c0-11e0-464c-a798-4cb6f334eb32" (UID: "debec1c0-11e0-464c-a798-4cb6f334eb32"). InnerVolumeSpecName "kube-api-access-9954z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:23.635159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.635102    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-token" pod "debec1c0-11e0-464c-a798-4cb6f334eb32" (UID: "debec1c0-11e0-464c-a798-4cb6f334eb32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:23.733974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.733942    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.733974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.733971    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9954z\" (UniqueName: \"kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-kube-api-access-9954z\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.734371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.733983    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.734371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.733992    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.734371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.734001    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.734371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.734010    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:23.734371 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:23.734018    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debec1c0-11e0-464c-a798-4cb6f334eb32-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:24.438457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:24.438432    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"
Oct 30 21:55:24.438457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:24.438428    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6" event={"ID":"debec1c0-11e0-464c-a798-4cb6f334eb32","Type":"ContainerDied","Data":"6484e4f0afda6dbe2c0d3b70fe5fa42551a7bc4bf6f32a92df8badd3d86af239"}
Oct 30 21:55:24.438718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:24.438500    4918 scope.go:122] "RemoveContainer" containerID="d667bfc9190a02448fa011b20c8cdd7e61c86a3b3051e6bdc916969767794d1a"
Oct 30 21:55:24.464285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:24.464249    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"]
Oct 30 21:55:24.469793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:24.469770    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-15e79c-openshift-default-7cc47b99df-r87g6"]
Oct 30 21:55:25.078188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:25.078152    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="debec1c0-11e0-464c-a798-4cb6f334eb32" path="/var/lib/kubelet/pods/debec1c0-11e0-464c-a798-4cb6f334eb32/volumes"
Oct 30 21:55:25.725626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:25.725591    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"]
Oct 30 21:55:25.739769 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:25.739738    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"]
Oct 30 21:55:26.251276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:26.251232    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f" podUID="755a5f2c-b99e-4c7b-893a-d346b9f92d81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:26.251896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:26.251480    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc" podUID="4db3211f-c998-428e-9364-88befce1cf71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:27.274413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:27.273465    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"]
Oct 30 21:55:27.280682 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:27.278634    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-adf308-8959x"]
Oct 30 21:55:29.104305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:29.104260    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b82edd-4669-4e18-87b8-9aba34e2ca1d" path="/var/lib/kubelet/pods/28b82edd-4669-4e18-87b8-9aba34e2ca1d/volumes"
Oct 30 21:55:29.164498 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:29.164458    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:55:30.221145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:30.221111    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"]
Oct 30 21:55:30.224711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:30.224685    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a9230f-c4xxr"]
Oct 30 21:55:30.726533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:30.726487    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd" podUID="2c8d84c6-9d80-432e-bdc4-8e3684a473f8" containerName="payload-processing" containerID="cri-o://278df32917c4ad613c26799354b6f5f77982a37833ed555fd21023f3391b253f" gracePeriod=25
Oct 30 21:55:30.740782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:30.740748    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6" podUID="f5bf3460-e153-45ef-86db-837af9418a7d" containerName="payload-pre-processing" containerID="cri-o://e0218e50b8a4cca9d8f18159be80ff6c201ff7549169c1242a2abf029108972d" gracePeriod=25
Oct 30 21:55:31.079731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:31.079639    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de823b54-4177-44d2-b352-c98d6afde970" path="/var/lib/kubelet/pods/de823b54-4177-44d2-b352-c98d6afde970/volumes"
Oct 30 21:55:31.478855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:31.478823    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9" podUID="4515aee4-3a67-43d0-9d60-7ad6524bf8d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:31.479246 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:31.478881    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5" podUID="22a44e09-93c1-4e78-bc3e-52de37538ec3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:31.592121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:31.592085    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"]
Oct 30 21:55:31.592315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:31.592289    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy" containerID="cri-o://f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6" gracePeriod=30
Oct 30 21:55:32.005420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.005363    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"
Oct 30 21:55:32.069131 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.069099    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"
Oct 30 21:55:32.078342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.078322    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-plugins-config-volume\") pod \"4515aee4-3a67-43d0-9d60-7ad6524bf8d3\" (UID: \"4515aee4-3a67-43d0-9d60-7ad6524bf8d3\") "
Oct 30 21:55:32.078509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.078496    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-kube-api-access-qqcft\" (UniqueName: \"kubernetes.io/projected/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-kube-api-access-qqcft\") pod \"4515aee4-3a67-43d0-9d60-7ad6524bf8d3\" (UID: \"4515aee4-3a67-43d0-9d60-7ad6524bf8d3\") "
Oct 30 21:55:32.078747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.078722    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-plugins-config-volume" pod "4515aee4-3a67-43d0-9d60-7ad6524bf8d3" (UID: "4515aee4-3a67-43d0-9d60-7ad6524bf8d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:32.079101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.079085    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:32.080711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.080679    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-kube-api-access-qqcft" pod "4515aee4-3a67-43d0-9d60-7ad6524bf8d3" (UID: "4515aee4-3a67-43d0-9d60-7ad6524bf8d3"). InnerVolumeSpecName "kube-api-access-qqcft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:32.161235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.161202    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"]
Oct 30 21:55:32.161909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.161892    4918 cpu_manager.go:414] "containerMap: removing container" podUID="de823b54-4177-44d2-b352-c98d6afde970" containerName="revoke-keys"
Oct 30 21:55:32.161968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.161911    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de823b54-4177-44d2-b352-c98d6afde970" containerName="revoke-keys"
Oct 30 21:55:32.161968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.161931    4918 cpu_manager.go:414] "containerMap: removing container" podUID="debec1c0-11e0-464c-a798-4cb6f334eb32" containerName="istio-proxy"
Oct 30 21:55:32.161968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.161937    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debec1c0-11e0-464c-a798-4cb6f334eb32" containerName="istio-proxy"
Oct 30 21:55:32.161968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.161963    4918 cpu_manager.go:414] "containerMap: removing container" podUID="22a44e09-93c1-4e78-bc3e-52de37538ec3" containerName="payload-pre-processing"
Oct 30 21:55:32.161968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.161968    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a44e09-93c1-4e78-bc3e-52de37538ec3" containerName="payload-pre-processing"
Oct 30 21:55:32.162138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.161983    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4515aee4-3a67-43d0-9d60-7ad6524bf8d3" containerName="payload-processing"
Oct 30 21:55:32.162138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.161988    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4515aee4-3a67-43d0-9d60-7ad6524bf8d3" containerName="payload-processing"
Oct 30 21:55:32.162138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.162003    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" containerName="istio-proxy"
Oct 30 21:55:32.162138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.162007    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" containerName="istio-proxy"
Oct 30 21:55:32.162138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.162092    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4515aee4-3a67-43d0-9d60-7ad6524bf8d3" containerName="payload-processing"
Oct 30 21:55:32.162138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.162101    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ddd4fc4-2a3a-450b-898b-81d92045fc65" containerName="istio-proxy"
Oct 30 21:55:32.162138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.162110    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="de823b54-4177-44d2-b352-c98d6afde970" containerName="revoke-keys"
Oct 30 21:55:32.162138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.162118    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="debec1c0-11e0-464c-a798-4cb6f334eb32" containerName="istio-proxy"
Oct 30 21:55:32.162138 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.162127    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a44e09-93c1-4e78-bc3e-52de37538ec3" containerName="payload-pre-processing"
Oct 30 21:55:32.165800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.165785    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:32.176943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.176921    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:55:32.176943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.176921    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:55:32.180256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.180234    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"]
Oct 30 21:55:32.180703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.180675    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a44e09-93c1-4e78-bc3e-52de37538ec3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a44e09-93c1-4e78-bc3e-52de37538ec3-plugins-config-volume\") pod \"22a44e09-93c1-4e78-bc3e-52de37538ec3\" (UID: \"22a44e09-93c1-4e78-bc3e-52de37538ec3\") "
Oct 30 21:55:32.180789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.180719    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a44e09-93c1-4e78-bc3e-52de37538ec3-kube-api-access-cqxn9\" (UniqueName: \"kubernetes.io/projected/22a44e09-93c1-4e78-bc3e-52de37538ec3-kube-api-access-cqxn9\") pod \"22a44e09-93c1-4e78-bc3e-52de37538ec3\" (UID: \"22a44e09-93c1-4e78-bc3e-52de37538ec3\") "
Oct 30 21:55:32.181048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.181026    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd\" (UID: \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:32.181148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.181053    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a44e09-93c1-4e78-bc3e-52de37538ec3-plugins-config-volume" pod "22a44e09-93c1-4e78-bc3e-52de37538ec3" (UID: "22a44e09-93c1-4e78-bc3e-52de37538ec3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:32.181422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.181362    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nswc2\" (UniqueName: \"kubernetes.io/projected/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-kube-api-access-nswc2\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd\" (UID: \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:32.181515 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.181449    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd\" (UID: \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:32.181943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.181919    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqcft\" (UniqueName: \"kubernetes.io/projected/4515aee4-3a67-43d0-9d60-7ad6524bf8d3-kube-api-access-qqcft\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:32.182031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.181947    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22a44e09-93c1-4e78-bc3e-52de37538ec3-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:32.183109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.183085    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a44e09-93c1-4e78-bc3e-52de37538ec3-kube-api-access-cqxn9" pod "22a44e09-93c1-4e78-bc3e-52de37538ec3" (UID: "22a44e09-93c1-4e78-bc3e-52de37538ec3"). InnerVolumeSpecName "kube-api-access-cqxn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:32.283099 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.283061    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nswc2\" (UniqueName: \"kubernetes.io/projected/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-kube-api-access-nswc2\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd\" (UID: \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:32.283303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.283113    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd\" (UID: \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:32.283303 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.283288    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd\" (UID: \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:32.283570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.283553    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqxn9\" (UniqueName: \"kubernetes.io/projected/22a44e09-93c1-4e78-bc3e-52de37538ec3-kube-api-access-cqxn9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:32.283923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.283900    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd\" (UID: \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:32.285824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.285794    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd\" (UID: \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:32.292124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.292101    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nswc2\" (UniqueName: \"kubernetes.io/projected/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-kube-api-access-nswc2\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd\" (UID: \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:32.480439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.480410    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:32.571415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.568511    4918 generic.go:356] "Generic (PLEG): container finished" podID="22a44e09-93c1-4e78-bc3e-52de37538ec3" containerID="526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9" exitCode=0
Oct 30 21:55:32.571415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.568699    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5" event={"ID":"22a44e09-93c1-4e78-bc3e-52de37538ec3","Type":"ContainerDied","Data":"526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9"}
Oct 30 21:55:32.571415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.568785    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5" event={"ID":"22a44e09-93c1-4e78-bc3e-52de37538ec3","Type":"ContainerDied","Data":"45842a3d9dbb9a39a8cbfeba160c450d73688d1a51020b3a00e9ff3c852f0b2b"}
Oct 30 21:55:32.571415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.568815    4918 scope.go:122] "RemoveContainer" containerID="526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9"
Oct 30 21:55:32.571415 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.568992    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"
Oct 30 21:55:32.574869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.574845    4918 generic.go:356] "Generic (PLEG): container finished" podID="4515aee4-3a67-43d0-9d60-7ad6524bf8d3" containerID="e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d" exitCode=0
Oct 30 21:55:32.575010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.574881    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9" event={"ID":"4515aee4-3a67-43d0-9d60-7ad6524bf8d3","Type":"ContainerDied","Data":"e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d"}
Oct 30 21:55:32.575010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.574920    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9" event={"ID":"4515aee4-3a67-43d0-9d60-7ad6524bf8d3","Type":"ContainerDied","Data":"9af1d3afc5075fb44b238eb26cf1b5b1df576631b0c1d9a57817342089924500"}
Oct 30 21:55:32.575010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.574987    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"
Oct 30 21:55:32.602539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.602512    4918 scope.go:122] "RemoveContainer" containerID="526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9"
Oct 30 21:55:32.603016 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:32.602990    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9\": container with ID starting with 526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9 not found: ID does not exist" containerID="526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9"
Oct 30 21:55:32.603127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.603024    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9"} err="failed to get container status \"526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9\": rpc error: code = NotFound desc = could not find container \"526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9\": container with ID starting with 526d0890cd4bc3e24da73719cdb079f3e240c61266ecde5d3236aa84557a8ca9 not found: ID does not exist"
Oct 30 21:55:32.603127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.603045    4918 scope.go:122] "RemoveContainer" containerID="e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d"
Oct 30 21:55:32.617294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.617273    4918 scope.go:122] "RemoveContainer" containerID="e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d"
Oct 30 21:55:32.617696 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:32.617669    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d\": container with ID starting with e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d not found: ID does not exist" containerID="e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d"
Oct 30 21:55:32.617798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.617707    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d"} err="failed to get container status \"e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d\": rpc error: code = NotFound desc = could not find container \"e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d\": container with ID starting with e5b3350ac4917240f61d5e54a14aa6e588b1d3ad7f40628483d83a1b0a895b8d not found: ID does not exist"
Oct 30 21:55:32.623321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.623271    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"]
Oct 30 21:55:32.629312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.629291    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-726a18-cfd7c888c-hs6r5"]
Oct 30 21:55:32.635247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.635128    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"]
Oct 30 21:55:32.639048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.639029    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-726a18-7b8d9dd49f-tmwt9"]
Oct 30 21:55:32.789664 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:55:32.789635    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b7f23a_6935_41cf_ad55_85f0b7cc4a42.slice/crio-34da4b0a0085661d3d6cc46dc4fc18c9fcacde73173672f4b4561742a5cbebc2 WatchSource:0}: Error finding container 34da4b0a0085661d3d6cc46dc4fc18c9fcacde73173672f4b4561742a5cbebc2: Status 404 returned error can't find the container with id 34da4b0a0085661d3d6cc46dc4fc18c9fcacde73173672f4b4561742a5cbebc2
Oct 30 21:55:32.796497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:32.796474    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"]
Oct 30 21:55:33.078447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:33.078319    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a44e09-93c1-4e78-bc3e-52de37538ec3" path="/var/lib/kubelet/pods/22a44e09-93c1-4e78-bc3e-52de37538ec3/volumes"
Oct 30 21:55:33.078928 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:33.078904    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4515aee4-3a67-43d0-9d60-7ad6524bf8d3" path="/var/lib/kubelet/pods/4515aee4-3a67-43d0-9d60-7ad6524bf8d3/volumes"
Oct 30 21:55:33.587146 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:33.587114    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"]
Oct 30 21:55:33.593058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:33.593030    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d4a0a5-sxgbs"]
Oct 30 21:55:33.597622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:33.597594    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd" event={"ID":"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42","Type":"ContainerStarted","Data":"603f76343d25dc9b54081881f8044235d3da61493b2ebe94e0b4d687f44e198c"}
Oct 30 21:55:33.597762 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:33.597643    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd" event={"ID":"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42","Type":"ContainerStarted","Data":"34da4b0a0085661d3d6cc46dc4fc18c9fcacde73173672f4b4561742a5cbebc2"}
Oct 30 21:55:33.621538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:33.621498    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd" podStartSLOduration=1.621476304 podStartE2EDuration="1.621476304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:33.618168291 +0000 UTC m=+3246.664727456" watchObservedRunningTime="2026-10-30 21:55:33.621476304 +0000 UTC m=+3246.668035448"
Oct 30 21:55:34.607721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:34.607691    4918 generic.go:356] "Generic (PLEG): container finished" podID="e8b7f23a-6935-41cf-ad55-85f0b7cc4a42" containerID="603f76343d25dc9b54081881f8044235d3da61493b2ebe94e0b4d687f44e198c" exitCode=0
Oct 30 21:55:34.608155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:34.607782    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd" event={"ID":"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42","Type":"ContainerDied","Data":"603f76343d25dc9b54081881f8044235d3da61493b2ebe94e0b4d687f44e198c"}
Oct 30 21:55:35.078155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.078124    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208a3de5-1523-4585-8569-f8314c1d2810" path="/var/lib/kubelet/pods/208a3de5-1523-4585-8569-f8314c1d2810/volumes"
Oct 30 21:55:35.106078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.106040    4918 scope.go:122] "RemoveContainer" containerID="0032c21601142c952da06f52618680406b9f93f123defcac8bedc41064eaf9b0"
Oct 30 21:55:35.117849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.117829    4918 scope.go:122] "RemoveContainer" containerID="28c81b41f0be9d7a76f06fea787f7ec8014748a84f63cb1db4e50e2936c0cb76"
Oct 30 21:55:35.131090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.131066    4918 scope.go:122] "RemoveContainer" containerID="97be1ef4748988ec0b65fce340430d412cbe17beeef83618037c04fff971de38"
Oct 30 21:55:35.143658 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.143627    4918 scope.go:122] "RemoveContainer" containerID="2cc5c9ad33d7a5c2e85890354b544d0dca4a2b7cee179e04a12add93c5cb0d21"
Oct 30 21:55:35.388305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.387527    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6" podUID="f5bf3460-e153-45ef-86db-837af9418a7d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:35.736715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.736684    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"]
Oct 30 21:55:35.743544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.743515    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:35.751040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.750824    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"]
Oct 30 21:55:35.831035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.831012    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:35.867482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.867447    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nflhp\" (UniqueName: \"kubernetes.io/projected/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-kube-api-access-nflhp\") pod \"maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv\" (UID: \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:35.868520 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.868382    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv\" (UID: \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:35.868703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.868682    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv\" (UID: \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:35.981464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.981034    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-xks-ca\") pod \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\" (UID: \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\") "
Oct 30 21:55:35.981464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.981295    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-openshift-ca\") pod \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\" (UID: \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\") "
Oct 30 21:55:35.981464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.981349    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-kube-api-access-nswc2\" (UniqueName: \"kubernetes.io/projected/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-kube-api-access-nswc2\") pod \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\" (UID: \"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42\") "
Oct 30 21:55:35.981734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.981666    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv\" (UID: \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:35.987462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.981865    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv\" (UID: \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:35.987462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.981943    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nflhp\" (UniqueName: \"kubernetes.io/projected/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-kube-api-access-nflhp\") pod \"maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv\" (UID: \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:35.997423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.994361    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-openshift-ca" pod "e8b7f23a-6935-41cf-ad55-85f0b7cc4a42" (UID: "e8b7f23a-6935-41cf-ad55-85f0b7cc4a42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:35.997423 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:35.995517    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv\" (UID: \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:36.018818 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.018786    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv\" (UID: \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:36.019864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.019624    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nflhp\" (UniqueName: \"kubernetes.io/projected/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-kube-api-access-nflhp\") pod \"maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv\" (UID: \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:36.020326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.020303    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-xks-ca" pod "e8b7f23a-6935-41cf-ad55-85f0b7cc4a42" (UID: "e8b7f23a-6935-41cf-ad55-85f0b7cc4a42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:36.046667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.046634    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-kube-api-access-nswc2" pod "e8b7f23a-6935-41cf-ad55-85f0b7cc4a42" (UID: "e8b7f23a-6935-41cf-ad55-85f0b7cc4a42"). InnerVolumeSpecName "kube-api-access-nswc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:36.074943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.071939    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:36.083334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.083301    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:36.083486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.083340    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nswc2\" (UniqueName: \"kubernetes.io/projected/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-kube-api-access-nswc2\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:36.083486 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.083357    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:36.252473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.251739    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc" podUID="4db3211f-c998-428e-9364-88befce1cf71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:36.256416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.255084    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f" podUID="755a5f2c-b99e-4c7b-893a-d346b9f92d81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:36.358426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.357563    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd" podUID="2c8d84c6-9d80-432e-bdc4-8e3684a473f8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:36.507467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.507442    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"]
Oct 30 21:55:36.508057 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:55:36.508030    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cbfccc_c9b7_4861_a9a2_7e156ffbaf68.slice/crio-866aa272bdaed98f799d0a900fcba6cfab48880400e4c7eee81329873a9f6bc1 WatchSource:0}: Error finding container 866aa272bdaed98f799d0a900fcba6cfab48880400e4c7eee81329873a9f6bc1: Status 404 returned error can't find the container with id 866aa272bdaed98f799d0a900fcba6cfab48880400e4c7eee81329873a9f6bc1
Oct 30 21:55:36.638739 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.638703    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv" event={"ID":"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68","Type":"ContainerStarted","Data":"866aa272bdaed98f799d0a900fcba6cfab48880400e4c7eee81329873a9f6bc1"}
Oct 30 21:55:36.640165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.640140    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"
Oct 30 21:55:36.640289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.640158    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd" event={"ID":"e8b7f23a-6935-41cf-ad55-85f0b7cc4a42","Type":"ContainerDied","Data":"34da4b0a0085661d3d6cc46dc4fc18c9fcacde73173672f4b4561742a5cbebc2"}
Oct 30 21:55:36.640289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.640207    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34da4b0a0085661d3d6cc46dc4fc18c9fcacde73173672f4b4561742a5cbebc2"
Oct 30 21:55:36.881871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:36.881843    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:55:37.008781 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.008750    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-kube-api-access-5vq6t\" (UniqueName: \"kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-kube-api-access-5vq6t\") pod \"b524a7be-3268-4672-a309-a4888c48cf87\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") "
Oct 30 21:55:37.008982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.008869    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b524a7be-3268-4672-a309-a4888c48cf87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b524a7be-3268-4672-a309-a4888c48cf87-istio-podinfo\") pod \"b524a7be-3268-4672-a309-a4888c48cf87\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") "
Oct 30 21:55:37.008982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.008934    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-socket\") pod \"b524a7be-3268-4672-a309-a4888c48cf87\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") "
Oct 30 21:55:37.008982 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.008962    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-credential-socket\") pod \"b524a7be-3268-4672-a309-a4888c48cf87\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") "
Oct 30 21:55:37.009171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.009009    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b524a7be-3268-4672-a309-a4888c48cf87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b524a7be-3268-4672-a309-a4888c48cf87-istiod-ca-cert\") pod \"b524a7be-3268-4672-a309-a4888c48cf87\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") "
Oct 30 21:55:37.009171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.009091    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-envoy\") pod \"b524a7be-3268-4672-a309-a4888c48cf87\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") "
Oct 30 21:55:37.009171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.009114    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-data\") pod \"b524a7be-3268-4672-a309-a4888c48cf87\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") "
Oct 30 21:55:37.009351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.009245    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-certs\") pod \"b524a7be-3268-4672-a309-a4888c48cf87\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") "
Oct 30 21:55:37.009351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.009276    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-socket" pod "b524a7be-3268-4672-a309-a4888c48cf87" (UID: "b524a7be-3268-4672-a309-a4888c48cf87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:37.009351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.009320    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-istio-token\" (UniqueName: \"kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-istio-token\") pod \"b524a7be-3268-4672-a309-a4888c48cf87\" (UID: \"b524a7be-3268-4672-a309-a4888c48cf87\") "
Oct 30 21:55:37.009569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.009437    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-data" pod "b524a7be-3268-4672-a309-a4888c48cf87" (UID: "b524a7be-3268-4672-a309-a4888c48cf87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:37.009569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.009543    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-credential-socket" pod "b524a7be-3268-4672-a309-a4888c48cf87" (UID: "b524a7be-3268-4672-a309-a4888c48cf87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:37.009864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.009581    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-certs" pod "b524a7be-3268-4672-a309-a4888c48cf87" (UID: "b524a7be-3268-4672-a309-a4888c48cf87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:37.009864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.009631    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b524a7be-3268-4672-a309-a4888c48cf87-istiod-ca-cert" pod "b524a7be-3268-4672-a309-a4888c48cf87" (UID: "b524a7be-3268-4672-a309-a4888c48cf87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:37.010249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.010223    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.010341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.010255    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.010341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.010272    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b524a7be-3268-4672-a309-a4888c48cf87-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.010341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.010286    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.010341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.010299    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.011487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.011460    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-kube-api-access-5vq6t" pod "b524a7be-3268-4672-a309-a4888c48cf87" (UID: "b524a7be-3268-4672-a309-a4888c48cf87"). InnerVolumeSpecName "kube-api-access-5vq6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:37.011594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.011566    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b524a7be-3268-4672-a309-a4888c48cf87-istio-podinfo" pod "b524a7be-3268-4672-a309-a4888c48cf87" (UID: "b524a7be-3268-4672-a309-a4888c48cf87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:55:37.011661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.011574    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-envoy" pod "b524a7be-3268-4672-a309-a4888c48cf87" (UID: "b524a7be-3268-4672-a309-a4888c48cf87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:37.011790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.011774    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-istio-token" pod "b524a7be-3268-4672-a309-a4888c48cf87" (UID: "b524a7be-3268-4672-a309-a4888c48cf87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:37.112040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.112008    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b524a7be-3268-4672-a309-a4888c48cf87-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.112040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.112039    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.112253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.112051    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vq6t\" (UniqueName: \"kubernetes.io/projected/b524a7be-3268-4672-a309-a4888c48cf87-kube-api-access-5vq6t\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.112253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.112062    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b524a7be-3268-4672-a309-a4888c48cf87-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:37.679506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.677178    4918 generic.go:356] "Generic (PLEG): container finished" podID="b524a7be-3268-4672-a309-a4888c48cf87" containerID="f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6" exitCode=0
Oct 30 21:55:37.679506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.677511    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" event={"ID":"b524a7be-3268-4672-a309-a4888c48cf87","Type":"ContainerDied","Data":"f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6"}
Oct 30 21:55:37.679506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.677568    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9" event={"ID":"b524a7be-3268-4672-a309-a4888c48cf87","Type":"ContainerDied","Data":"fa0414fc1caf3dc3cf76aaa56b560993b2ac454ef8f269bf0a2fac3177250127"}
Oct 30 21:55:37.679506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.677599    4918 scope.go:122] "RemoveContainer" containerID="f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6"
Oct 30 21:55:37.679506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.677757    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"
Oct 30 21:55:37.709601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.709280    4918 generic.go:356] "Generic (PLEG): container finished" podID="60cbfccc-c9b7-4861-a9a2-7e156ffbaf68" containerID="06c285d6993a3b54dd2f0c9068b6fa88d36393f40860a43af014c99073805620" exitCode=0
Oct 30 21:55:37.709601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.709349    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv" event={"ID":"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68","Type":"ContainerDied","Data":"06c285d6993a3b54dd2f0c9068b6fa88d36393f40860a43af014c99073805620"}
Oct 30 21:55:37.752411 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.745584    4918 scope.go:122] "RemoveContainer" containerID="95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1"
Oct 30 21:55:37.785493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.785454    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"]
Oct 30 21:55:37.790757 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.790727    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-726a18-openshift-default-57dd9fc85-pbwx9"]
Oct 30 21:55:37.816354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.816333    4918 scope.go:122] "RemoveContainer" containerID="f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6"
Oct 30 21:55:37.816719 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:37.816699    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6\": container with ID starting with f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6 not found: ID does not exist" containerID="f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6"
Oct 30 21:55:37.816793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.816731    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6"} err="failed to get container status \"f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6\": rpc error: code = NotFound desc = could not find container \"f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6\": container with ID starting with f69d4fc9bf73827bc7063b234944b482b7d057dbd81a84f8f659dde1619c59a6 not found: ID does not exist"
Oct 30 21:55:37.816793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.816753    4918 scope.go:122] "RemoveContainer" containerID="95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1"
Oct 30 21:55:37.816995 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:37.816978    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1\": container with ID starting with 95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1 not found: ID does not exist" containerID="95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1"
Oct 30 21:55:37.817038 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.817002    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1"} err="failed to get container status \"95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1\": rpc error: code = NotFound desc = could not find container \"95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1\": container with ID starting with 95bdba15f1badbbb6c196a06c49a3d5f117cafd076911e4b7a173116778515a1 not found: ID does not exist"
Oct 30 21:55:37.964156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.964071    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"]
Oct 30 21:55:37.964666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:37.964319    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerName="istio-proxy" containerID="cri-o://0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78" gracePeriod=30
Oct 30 21:55:38.587764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.587732    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"]
Oct 30 21:55:38.588346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.588333    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy"
Oct 30 21:55:38.588426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.588348    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy"
Oct 30 21:55:38.588426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.588369    4918 cpu_manager.go:414] "containerMap: removing container" podUID="e8b7f23a-6935-41cf-ad55-85f0b7cc4a42" containerName="revoke-keys"
Oct 30 21:55:38.588426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.588376    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b7f23a-6935-41cf-ad55-85f0b7cc4a42" containerName="revoke-keys"
Oct 30 21:55:38.588426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.588405    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy"
Oct 30 21:55:38.588426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.588414    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy"
Oct 30 21:55:38.588610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.588530    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy"
Oct 30 21:55:38.588610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.588541    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b7f23a-6935-41cf-ad55-85f0b7cc4a42" containerName="revoke-keys"
Oct 30 21:55:38.588610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.588548    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b524a7be-3268-4672-a309-a4888c48cf87" containerName="istio-proxy"
Oct 30 21:55:38.593709 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.593687    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:38.603433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.602154    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"]
Oct 30 21:55:38.738477 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.738450    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd\" (UID: \"459a5db7-a125-41cb-91c0-8493b9748fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:38.738703 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.738675    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd\" (UID: \"459a5db7-a125-41cb-91c0-8493b9748fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:38.738784 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.738769    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvw9g\" (UniqueName: \"kubernetes.io/projected/459a5db7-a125-41cb-91c0-8493b9748fd0-kube-api-access-fvw9g\") pod \"maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd\" (UID: \"459a5db7-a125-41cb-91c0-8493b9748fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:38.840647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.840567    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd\" (UID: \"459a5db7-a125-41cb-91c0-8493b9748fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:38.840647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.840637    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvw9g\" (UniqueName: \"kubernetes.io/projected/459a5db7-a125-41cb-91c0-8493b9748fd0-kube-api-access-fvw9g\") pod \"maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd\" (UID: \"459a5db7-a125-41cb-91c0-8493b9748fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:38.840951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.840795    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd\" (UID: \"459a5db7-a125-41cb-91c0-8493b9748fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:38.842098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.842070    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd\" (UID: \"459a5db7-a125-41cb-91c0-8493b9748fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:38.844076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.844053    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd\" (UID: \"459a5db7-a125-41cb-91c0-8493b9748fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:38.858500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.858480    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvw9g\" (UniqueName: \"kubernetes.io/projected/459a5db7-a125-41cb-91c0-8493b9748fd0-kube-api-access-fvw9g\") pod \"maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd\" (UID: \"459a5db7-a125-41cb-91c0-8493b9748fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:38.888907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.888881    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:38.910589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:38.910562    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:39.043851 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.043820    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-kube-api-access-nflhp\" (UniqueName: \"kubernetes.io/projected/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-kube-api-access-nflhp\") pod \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\" (UID: \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\") "
Oct 30 21:55:39.044345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.043959    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-xks-ca\") pod \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\" (UID: \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\") "
Oct 30 21:55:39.044345 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.044024    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-openshift-ca\") pod \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\" (UID: \"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68\") "
Oct 30 21:55:39.044619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.044589    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-openshift-ca" pod "60cbfccc-c9b7-4861-a9a2-7e156ffbaf68" (UID: "60cbfccc-c9b7-4861-a9a2-7e156ffbaf68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:39.044782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.044718    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:39.046407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.046371    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-xks-ca" pod "60cbfccc-c9b7-4861-a9a2-7e156ffbaf68" (UID: "60cbfccc-c9b7-4861-a9a2-7e156ffbaf68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:39.047002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.046984    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-kube-api-access-nflhp" pod "60cbfccc-c9b7-4861-a9a2-7e156ffbaf68" (UID: "60cbfccc-c9b7-4861-a9a2-7e156ffbaf68"). InnerVolumeSpecName "kube-api-access-nflhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:39.078838 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.078803    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b524a7be-3268-4672-a309-a4888c48cf87" path="/var/lib/kubelet/pods/b524a7be-3268-4672-a309-a4888c48cf87/volumes"
Oct 30 21:55:39.146409 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.146322    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nflhp\" (UniqueName: \"kubernetes.io/projected/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-kube-api-access-nflhp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:39.146409 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.146346    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:39.209118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.209093    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"]
Oct 30 21:55:39.209673 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:55:39.209650    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459a5db7_a125_41cb_91c0_8493b9748fd0.slice/crio-4514be6f750d413dcd19867a2868618d288df63f7f3e6bf404a81dd4a9dff3ba WatchSource:0}: Error finding container 4514be6f750d413dcd19867a2868618d288df63f7f3e6bf404a81dd4a9dff3ba: Status 404 returned error can't find the container with id 4514be6f750d413dcd19867a2868618d288df63f7f3e6bf404a81dd4a9dff3ba
Oct 30 21:55:39.726292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.726266    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"
Oct 30 21:55:39.726481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.726292    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv" event={"ID":"60cbfccc-c9b7-4861-a9a2-7e156ffbaf68","Type":"ContainerDied","Data":"866aa272bdaed98f799d0a900fcba6cfab48880400e4c7eee81329873a9f6bc1"}
Oct 30 21:55:39.726481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.726370    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="866aa272bdaed98f799d0a900fcba6cfab48880400e4c7eee81329873a9f6bc1"
Oct 30 21:55:39.728054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.728027    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd" event={"ID":"459a5db7-a125-41cb-91c0-8493b9748fd0","Type":"ContainerStarted","Data":"805883bfec201c87707656ad706cc436bb04c2e1bee6687dabf84db6c8834f51"}
Oct 30 21:55:39.728184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.728072    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd" event={"ID":"459a5db7-a125-41cb-91c0-8493b9748fd0","Type":"ContainerStarted","Data":"4514be6f750d413dcd19867a2868618d288df63f7f3e6bf404a81dd4a9dff3ba"}
Oct 30 21:55:39.749810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:39.749763    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd" podStartSLOduration=1.749749875 podStartE2EDuration="1.749749875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:39.746501326 +0000 UTC m=+3252.793060483" watchObservedRunningTime="2026-10-30 21:55:39.749749875 +0000 UTC m=+3252.796309017"
Oct 30 21:55:40.737335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:40.737305    4918 generic.go:356] "Generic (PLEG): container finished" podID="459a5db7-a125-41cb-91c0-8493b9748fd0" containerID="805883bfec201c87707656ad706cc436bb04c2e1bee6687dabf84db6c8834f51" exitCode=0
Oct 30 21:55:40.737753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:40.737421    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd" event={"ID":"459a5db7-a125-41cb-91c0-8493b9748fd0","Type":"ContainerDied","Data":"805883bfec201c87707656ad706cc436bb04c2e1bee6687dabf84db6c8834f51"}
Oct 30 21:55:41.107006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.106927    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"]
Oct 30 21:55:41.107156 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.107098    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" podUID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" containerName="istio-proxy" containerID="cri-o://ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af" gracePeriod=30
Oct 30 21:55:41.639172 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.639136    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"]
Oct 30 21:55:41.639828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.639810    4918 cpu_manager.go:414] "containerMap: removing container" podUID="60cbfccc-c9b7-4861-a9a2-7e156ffbaf68" containerName="revoke-keys"
Oct 30 21:55:41.639905 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.639830    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60cbfccc-c9b7-4861-a9a2-7e156ffbaf68" containerName="revoke-keys"
Oct 30 21:55:41.639957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.639948    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="60cbfccc-c9b7-4861-a9a2-7e156ffbaf68" containerName="revoke-keys"
Oct 30 21:55:41.643669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.643651    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:41.652295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.652270    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"]
Oct 30 21:55:41.798557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.795927    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r\" (UID: \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:41.798557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.796031    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r\" (UID: \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:41.798557 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.796193    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmmt\" (UniqueName: \"kubernetes.io/projected/ee87d020-7e1e-4f32-a2ab-3615215f49bf-kube-api-access-8lmmt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r\" (UID: \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:41.897596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.897525    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmmt\" (UniqueName: \"kubernetes.io/projected/ee87d020-7e1e-4f32-a2ab-3615215f49bf-kube-api-access-8lmmt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r\" (UID: \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:41.897775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.897691    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r\" (UID: \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:41.897775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.897757    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r\" (UID: \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:41.898621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.898590    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r\" (UID: \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:41.900812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.900783    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r\" (UID: \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:41.907704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.907682    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmmt\" (UniqueName: \"kubernetes.io/projected/ee87d020-7e1e-4f32-a2ab-3615215f49bf-kube-api-access-8lmmt\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r\" (UID: \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:41.937114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.937095    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:41.960449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:41.960413    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:42.101232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.101194    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-xks-ca\") pod \"459a5db7-a125-41cb-91c0-8493b9748fd0\" (UID: \"459a5db7-a125-41cb-91c0-8493b9748fd0\") "
Oct 30 21:55:42.101453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.101288    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-openshift-ca\") pod \"459a5db7-a125-41cb-91c0-8493b9748fd0\" (UID: \"459a5db7-a125-41cb-91c0-8493b9748fd0\") "
Oct 30 21:55:42.101453 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.101361    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459a5db7-a125-41cb-91c0-8493b9748fd0-kube-api-access-fvw9g\" (UniqueName: \"kubernetes.io/projected/459a5db7-a125-41cb-91c0-8493b9748fd0-kube-api-access-fvw9g\") pod \"459a5db7-a125-41cb-91c0-8493b9748fd0\" (UID: \"459a5db7-a125-41cb-91c0-8493b9748fd0\") "
Oct 30 21:55:42.101742 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.101713    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-openshift-ca" pod "459a5db7-a125-41cb-91c0-8493b9748fd0" (UID: "459a5db7-a125-41cb-91c0-8493b9748fd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:42.101959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.101943    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:42.103468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.103446    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-xks-ca" pod "459a5db7-a125-41cb-91c0-8493b9748fd0" (UID: "459a5db7-a125-41cb-91c0-8493b9748fd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:42.103618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.103602    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459a5db7-a125-41cb-91c0-8493b9748fd0-kube-api-access-fvw9g" pod "459a5db7-a125-41cb-91c0-8493b9748fd0" (UID: "459a5db7-a125-41cb-91c0-8493b9748fd0"). InnerVolumeSpecName "kube-api-access-fvw9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:42.203650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.203578    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvw9g\" (UniqueName: \"kubernetes.io/projected/459a5db7-a125-41cb-91c0-8493b9748fd0-kube-api-access-fvw9g\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:42.203650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.203606    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a5db7-a125-41cb-91c0-8493b9748fd0-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:42.264717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.264692    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"]
Oct 30 21:55:42.265190 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:55:42.265169    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee87d020_7e1e_4f32_a2ab_3615215f49bf.slice/crio-1ff7a206087b83668e92043d15068eeef637cb24aea5f4273f4370cd1e2539d9 WatchSource:0}: Error finding container 1ff7a206087b83668e92043d15068eeef637cb24aea5f4273f4370cd1e2539d9: Status 404 returned error can't find the container with id 1ff7a206087b83668e92043d15068eeef637cb24aea5f4273f4370cd1e2539d9
Oct 30 21:55:42.314264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.314236    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"]
Oct 30 21:55:42.374858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.374825    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"]
Oct 30 21:55:42.769726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.769691    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd" event={"ID":"459a5db7-a125-41cb-91c0-8493b9748fd0","Type":"ContainerDied","Data":"4514be6f750d413dcd19867a2868618d288df63f7f3e6bf404a81dd4a9dff3ba"}
Oct 30 21:55:42.770514 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.769732    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"
Oct 30 21:55:42.770965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.770944    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4514be6f750d413dcd19867a2868618d288df63f7f3e6bf404a81dd4a9dff3ba"
Oct 30 21:55:42.772835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:42.772807    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r" event={"ID":"ee87d020-7e1e-4f32-a2ab-3615215f49bf","Type":"ContainerStarted","Data":"1ff7a206087b83668e92043d15068eeef637cb24aea5f4273f4370cd1e2539d9"}
Oct 30 21:55:43.245492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.245464    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:55:43.427436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.427368    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-kube-api-access-j4zjv\" (UniqueName: \"kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-kube-api-access-j4zjv\") pod \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") "
Oct 30 21:55:43.427558 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.427528    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-credential-socket\") pod \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") "
Oct 30 21:55:43.427728 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.427706    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istiod-ca-cert\") pod \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") "
Oct 30 21:55:43.427839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.427799    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-token\" (UniqueName: \"kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-token\") pod \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") "
Oct 30 21:55:43.427839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.427823    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-credential-socket" pod "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" (UID: "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:43.427928 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.427831    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-envoy\") pod \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") "
Oct 30 21:55:43.427928 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.427892    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-socket\") pod \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") "
Oct 30 21:55:43.428018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.428002    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-data\") pod \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") "
Oct 30 21:55:43.428102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.428081    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-certs\") pod \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") "
Oct 30 21:55:43.428102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.428092    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-socket" pod "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" (UID: "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:43.428232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.428146    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istiod-ca-cert" pod "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" (UID: "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:43.428232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.428180    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-podinfo\") pod \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\" (UID: \"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79\") "
Oct 30 21:55:43.428359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.428289    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-data" pod "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" (UID: "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:43.428359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.428345    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-certs" pod "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" (UID: "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:43.429314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.429143    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:43.429314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.429171    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:43.429314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.429186    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:43.429314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.429200    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:43.429314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.429215    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:43.430113 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.430084    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-kube-api-access-j4zjv" pod "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" (UID: "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79"). InnerVolumeSpecName "kube-api-access-j4zjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:43.430382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.430359    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-envoy" pod "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" (UID: "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:43.430706 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.430675    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-token" pod "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" (UID: "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:43.431485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.431463    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-podinfo" pod "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" (UID: "d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:55:43.434462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.434442    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"]
Oct 30 21:55:43.530701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.530670    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:43.530847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.530825    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4zjv\" (UniqueName: \"kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-kube-api-access-j4zjv\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:43.530847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.530848    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:43.530979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.530858    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:43.699284 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.699257    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"
Oct 30 21:55:43.786581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.786554    4918 generic.go:356] "Generic (PLEG): container finished" podID="755a5f2c-b99e-4c7b-893a-d346b9f92d81" containerID="7fce3c830e08331123298305c4bec06d9de0b27b04eee3d3f16a7385d31b0c47" exitCode=0
Oct 30 21:55:43.786684 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.786631    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f" event={"ID":"755a5f2c-b99e-4c7b-893a-d346b9f92d81","Type":"ContainerDied","Data":"7fce3c830e08331123298305c4bec06d9de0b27b04eee3d3f16a7385d31b0c47"}
Oct 30 21:55:43.788500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.788478    4918 generic.go:356] "Generic (PLEG): container finished" podID="ee87d020-7e1e-4f32-a2ab-3615215f49bf" containerID="2ceeb97c49484e3eb673720bcf2b3ca80bd9a471a55b5715aa5cad1e4e4ef0af" exitCode=0
Oct 30 21:55:43.788649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.788556    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r" event={"ID":"ee87d020-7e1e-4f32-a2ab-3615215f49bf","Type":"ContainerDied","Data":"2ceeb97c49484e3eb673720bcf2b3ca80bd9a471a55b5715aa5cad1e4e4ef0af"}
Oct 30 21:55:43.790050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.790026    4918 generic.go:356] "Generic (PLEG): container finished" podID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerID="0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78" exitCode=0
Oct 30 21:55:43.790174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.790059    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" event={"ID":"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79","Type":"ContainerDied","Data":"0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78"}
Oct 30 21:55:43.790174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.790108    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664" event={"ID":"d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79","Type":"ContainerDied","Data":"c8481ee9bdd9b4c39c87a181f56b265c342c9213f6dad0b6a5c5b10ac56d31dd"}
Oct 30 21:55:43.790174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.790137    4918 scope.go:122] "RemoveContainer" containerID="0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78"
Oct 30 21:55:43.790174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.790065    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"
Oct 30 21:55:43.791891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.791865    4918 generic.go:356] "Generic (PLEG): container finished" podID="4db3211f-c998-428e-9364-88befce1cf71" containerID="bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b" exitCode=0
Oct 30 21:55:43.792046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.792024    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc" event={"ID":"4db3211f-c998-428e-9364-88befce1cf71","Type":"ContainerDied","Data":"bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b"}
Oct 30 21:55:43.792194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.792071    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc" event={"ID":"4db3211f-c998-428e-9364-88befce1cf71","Type":"ContainerDied","Data":"24668ed08aa960c2b8404f1c4d868f14e3c039710956e831370d2017dd111aa1"}
Oct 30 21:55:43.792194 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.792136    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"
Oct 30 21:55:43.796313 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.796294    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"
Oct 30 21:55:43.802380 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.802362    4918 scope.go:122] "RemoveContainer" containerID="0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78"
Oct 30 21:55:43.802688 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:43.802670    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78\": container with ID starting with 0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78 not found: ID does not exist" containerID="0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78"
Oct 30 21:55:43.802768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.802695    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78"} err="failed to get container status \"0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78\": rpc error: code = NotFound desc = could not find container \"0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78\": container with ID starting with 0b56ec926fa8a5ceb2f7b6da303c32b25e81cb5f346d9ef56b9d2a816e829f78 not found: ID does not exist"
Oct 30 21:55:43.802768 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.802711    4918 scope.go:122] "RemoveContainer" containerID="bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b"
Oct 30 21:55:43.812722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.812643    4918 scope.go:122] "RemoveContainer" containerID="bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b"
Oct 30 21:55:43.813005 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:43.812977    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b\": container with ID starting with bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b not found: ID does not exist" containerID="bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b"
Oct 30 21:55:43.813106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.813013    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b"} err="failed to get container status \"bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b\": rpc error: code = NotFound desc = could not find container \"bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b\": container with ID starting with bc48f7d1e1a36b19a8ab5ed91269ccac5c26f71844972ff3f0d83a40a94f798b not found: ID does not exist"
Oct 30 21:55:43.835763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.835742    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/755a5f2c-b99e-4c7b-893a-d346b9f92d81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755a5f2c-b99e-4c7b-893a-d346b9f92d81-plugins-config-volume\") pod \"755a5f2c-b99e-4c7b-893a-d346b9f92d81\" (UID: \"755a5f2c-b99e-4c7b-893a-d346b9f92d81\") "
Oct 30 21:55:43.835942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.835846    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db3211f-c998-428e-9364-88befce1cf71-kube-api-access-vgr94\" (UniqueName: \"kubernetes.io/projected/4db3211f-c998-428e-9364-88befce1cf71-kube-api-access-vgr94\") pod \"4db3211f-c998-428e-9364-88befce1cf71\" (UID: \"4db3211f-c998-428e-9364-88befce1cf71\") "
Oct 30 21:55:43.835942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.835922    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755a5f2c-b99e-4c7b-893a-d346b9f92d81-kube-api-access-jllrl\" (UniqueName: \"kubernetes.io/projected/755a5f2c-b99e-4c7b-893a-d346b9f92d81-kube-api-access-jllrl\") pod \"755a5f2c-b99e-4c7b-893a-d346b9f92d81\" (UID: \"755a5f2c-b99e-4c7b-893a-d346b9f92d81\") "
Oct 30 21:55:43.836081 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.835958    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db3211f-c998-428e-9364-88befce1cf71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db3211f-c998-428e-9364-88befce1cf71-plugins-config-volume\") pod \"4db3211f-c998-428e-9364-88befce1cf71\" (UID: \"4db3211f-c998-428e-9364-88befce1cf71\") "
Oct 30 21:55:43.836143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.836106    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755a5f2c-b99e-4c7b-893a-d346b9f92d81-plugins-config-volume" pod "755a5f2c-b99e-4c7b-893a-d346b9f92d81" (UID: "755a5f2c-b99e-4c7b-893a-d346b9f92d81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:43.836433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.836410    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db3211f-c998-428e-9364-88befce1cf71-plugins-config-volume" pod "4db3211f-c998-428e-9364-88befce1cf71" (UID: "4db3211f-c998-428e-9364-88befce1cf71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:43.836565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.836490    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db3211f-c998-428e-9364-88befce1cf71-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:43.836565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.836513    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/755a5f2c-b99e-4c7b-893a-d346b9f92d81-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:43.838214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.838188    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755a5f2c-b99e-4c7b-893a-d346b9f92d81-kube-api-access-jllrl" pod "755a5f2c-b99e-4c7b-893a-d346b9f92d81" (UID: "755a5f2c-b99e-4c7b-893a-d346b9f92d81"). InnerVolumeSpecName "kube-api-access-jllrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:43.838346 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.838243    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db3211f-c998-428e-9364-88befce1cf71-kube-api-access-vgr94" pod "4db3211f-c998-428e-9364-88befce1cf71" (UID: "4db3211f-c998-428e-9364-88befce1cf71"). InnerVolumeSpecName "kube-api-access-vgr94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:43.843894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.843874    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"]
Oct 30 21:55:43.847408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.847371    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a9230f-openshift-default-767df789fc-zd664"]
Oct 30 21:55:43.938018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.937980    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jllrl\" (UniqueName: \"kubernetes.io/projected/755a5f2c-b99e-4c7b-893a-d346b9f92d81-kube-api-access-jllrl\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:43.938018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:43.938013    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgr94\" (UniqueName: \"kubernetes.io/projected/4db3211f-c998-428e-9364-88befce1cf71-kube-api-access-vgr94\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:44.122690 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:44.122662    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"]
Oct 30 21:55:44.126233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:44.126207    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-adf308-7df67775d9-cvxcc"]
Oct 30 21:55:44.803121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:44.803095    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"
Oct 30 21:55:44.803121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:44.803089    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f" event={"ID":"755a5f2c-b99e-4c7b-893a-d346b9f92d81","Type":"ContainerDied","Data":"e52613bf720faa914fe32129b8b9bae1e7af494b702214f16d2e0f8b119350a2"}
Oct 30 21:55:44.803889 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:44.803194    4918 scope.go:122] "RemoveContainer" containerID="7fce3c830e08331123298305c4bec06d9de0b27b04eee3d3f16a7385d31b0c47"
Oct 30 21:55:44.833475 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:44.832309    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"]
Oct 30 21:55:44.838111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:44.838084    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-adf308-6ccbd7fbcf-fv85f"]
Oct 30 21:55:44.972042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:44.972019    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:45.050455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.050424    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-openshift-ca\") pod \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\" (UID: \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\") "
Oct 30 21:55:45.050649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.050586    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-xks-ca\") pod \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\" (UID: \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\") "
Oct 30 21:55:45.050649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.050642    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee87d020-7e1e-4f32-a2ab-3615215f49bf-kube-api-access-8lmmt\" (UniqueName: \"kubernetes.io/projected/ee87d020-7e1e-4f32-a2ab-3615215f49bf-kube-api-access-8lmmt\") pod \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\" (UID: \"ee87d020-7e1e-4f32-a2ab-3615215f49bf\") "
Oct 30 21:55:45.050772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.050747    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-openshift-ca" pod "ee87d020-7e1e-4f32-a2ab-3615215f49bf" (UID: "ee87d020-7e1e-4f32-a2ab-3615215f49bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:45.051087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.051068    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:45.052873 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.052853    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-xks-ca" pod "ee87d020-7e1e-4f32-a2ab-3615215f49bf" (UID: "ee87d020-7e1e-4f32-a2ab-3615215f49bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:45.052955 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.052855    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee87d020-7e1e-4f32-a2ab-3615215f49bf-kube-api-access-8lmmt" pod "ee87d020-7e1e-4f32-a2ab-3615215f49bf" (UID: "ee87d020-7e1e-4f32-a2ab-3615215f49bf"). InnerVolumeSpecName "kube-api-access-8lmmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:45.078999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.078972    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db3211f-c998-428e-9364-88befce1cf71" path="/var/lib/kubelet/pods/4db3211f-c998-428e-9364-88befce1cf71/volumes"
Oct 30 21:55:45.079421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.079406    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="755a5f2c-b99e-4c7b-893a-d346b9f92d81" path="/var/lib/kubelet/pods/755a5f2c-b99e-4c7b-893a-d346b9f92d81/volumes"
Oct 30 21:55:45.079849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.079836    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" path="/var/lib/kubelet/pods/d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79/volumes"
Oct 30 21:55:45.152118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.152093    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee87d020-7e1e-4f32-a2ab-3615215f49bf-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:45.152229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.152119    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lmmt\" (UniqueName: \"kubernetes.io/projected/ee87d020-7e1e-4f32-a2ab-3615215f49bf-kube-api-access-8lmmt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:45.216626 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.216588    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:55:45.216778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.216649    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" podUID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:55:45.346590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.346504    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6" podUID="f5bf3460-e153-45ef-86db-837af9418a7d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:45.811875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.811846    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"
Oct 30 21:55:45.811875 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.811872    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r" event={"ID":"ee87d020-7e1e-4f32-a2ab-3615215f49bf","Type":"ContainerDied","Data":"1ff7a206087b83668e92043d15068eeef637cb24aea5f4273f4370cd1e2539d9"}
Oct 30 21:55:45.812438 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.811920    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ff7a206087b83668e92043d15068eeef637cb24aea5f4273f4370cd1e2539d9"
Oct 30 21:55:45.873701 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:45.873667    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"]
Oct 30 21:55:46.361691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.359765    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd" podUID="2c8d84c6-9d80-432e-bdc4-8e3684a473f8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:46.455304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.455278    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:55:46.521319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.521287    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-kube-api-access-qp5sh\" (UniqueName: \"kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-kube-api-access-qp5sh\") pod \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") "
Oct 30 21:55:46.521509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.521347    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-credential-socket\") pod \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") "
Oct 30 21:55:46.521509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.521433    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istiod-ca-cert\") pod \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") "
Oct 30 21:55:46.521509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.521492    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-certs\") pod \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") "
Oct 30 21:55:46.521716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.521558    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-data\") pod \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") "
Oct 30 21:55:46.521716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.521588    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-podinfo\") pod \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") "
Oct 30 21:55:46.521716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.521660    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-credential-socket" pod "f4378181-49fc-4f6b-b9ca-6700bec0fbc8" (UID: "f4378181-49fc-4f6b-b9ca-6700bec0fbc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:46.521716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.521708    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-envoy\") pod \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") "
Oct 30 21:55:46.521953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.521750    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-token\" (UniqueName: \"kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-token\") pod \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") "
Oct 30 21:55:46.521953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.521796    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-socket\") pod \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\" (UID: \"f4378181-49fc-4f6b-b9ca-6700bec0fbc8\") "
Oct 30 21:55:46.521953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.521860    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-certs" pod "f4378181-49fc-4f6b-b9ca-6700bec0fbc8" (UID: "f4378181-49fc-4f6b-b9ca-6700bec0fbc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:46.522163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.522139    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istiod-ca-cert" pod "f4378181-49fc-4f6b-b9ca-6700bec0fbc8" (UID: "f4378181-49fc-4f6b-b9ca-6700bec0fbc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:46.522364 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.522345    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-data" pod "f4378181-49fc-4f6b-b9ca-6700bec0fbc8" (UID: "f4378181-49fc-4f6b-b9ca-6700bec0fbc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:46.522463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.522360    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.522463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.522376    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.522463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.522385    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.522646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.522605    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-socket" pod "f4378181-49fc-4f6b-b9ca-6700bec0fbc8" (UID: "f4378181-49fc-4f6b-b9ca-6700bec0fbc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:46.524798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.524719    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-podinfo" pod "f4378181-49fc-4f6b-b9ca-6700bec0fbc8" (UID: "f4378181-49fc-4f6b-b9ca-6700bec0fbc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:55:46.524798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.524745    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-token" pod "f4378181-49fc-4f6b-b9ca-6700bec0fbc8" (UID: "f4378181-49fc-4f6b-b9ca-6700bec0fbc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:46.525188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.525164    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-envoy" pod "f4378181-49fc-4f6b-b9ca-6700bec0fbc8" (UID: "f4378181-49fc-4f6b-b9ca-6700bec0fbc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:46.525264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.525212    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-kube-api-access-qp5sh" pod "f4378181-49fc-4f6b-b9ca-6700bec0fbc8" (UID: "f4378181-49fc-4f6b-b9ca-6700bec0fbc8"). InnerVolumeSpecName "kube-api-access-qp5sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:46.623329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.623294    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.623329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.623322    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.623329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.623331    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.623329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.623340    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.623641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.623348    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.623641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.623357    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp5sh\" (UniqueName: \"kubernetes.io/projected/f4378181-49fc-4f6b-b9ca-6700bec0fbc8-kube-api-access-qp5sh\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:46.822482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.822446    4918 generic.go:356] "Generic (PLEG): container finished" podID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" containerID="ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af" exitCode=0
Oct 30 21:55:46.822977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.822497    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"
Oct 30 21:55:46.822977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.822503    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" event={"ID":"f4378181-49fc-4f6b-b9ca-6700bec0fbc8","Type":"ContainerDied","Data":"ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af"}
Oct 30 21:55:46.822977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.822578    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n" event={"ID":"f4378181-49fc-4f6b-b9ca-6700bec0fbc8","Type":"ContainerDied","Data":"8774cf624ba44418240277953497a4a3619323e5a52e3a5d8734e185f003464f"}
Oct 30 21:55:46.822977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.822600    4918 scope.go:122] "RemoveContainer" containerID="ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af"
Oct 30 21:55:46.834253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.834237    4918 scope.go:122] "RemoveContainer" containerID="ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af"
Oct 30 21:55:46.834608 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:46.834591    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af\": container with ID starting with ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af not found: ID does not exist" containerID="ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af"
Oct 30 21:55:46.834683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.834615    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af"} err="failed to get container status \"ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af\": rpc error: code = NotFound desc = could not find container \"ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af\": container with ID starting with ee817c63942176849a6b6f74cfdd2f240ce95173bd608c8c1d49ec1f53b3b7af not found: ID does not exist"
Oct 30 21:55:46.857508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.857481    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"]
Oct 30 21:55:46.862449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:46.862415    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-d4a0a5-openshift-default-7d5b4c8759-kf58n"]
Oct 30 21:55:47.104544 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:47.092572    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" path="/var/lib/kubelet/pods/f4378181-49fc-4f6b-b9ca-6700bec0fbc8/volumes"
Oct 30 21:55:47.315259 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:47.315222    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd" podUID="893a5d07-10fd-4b2f-a4a0-db937338f5db" containerName="payload-processing" containerID="cri-o://ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136" gracePeriod=25
Oct 30 21:55:47.376207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:47.376110    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg" podUID="cc7098ea-8a7e-4286-89e2-c3f6bf14d92b" containerName="payload-pre-processing" containerID="cri-o://76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578" gracePeriod=25
Oct 30 21:55:48.824262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:48.824228    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"]
Oct 30 21:55:48.838363 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:48.838330    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"]
Oct 30 21:55:50.168330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:50.168297    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg" podUID="cc7098ea-8a7e-4286-89e2-c3f6bf14d92b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:50.870748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:50.870714    4918 generic.go:356] "Generic (PLEG): container finished" podID="f5bf3460-e153-45ef-86db-837af9418a7d" containerID="e0218e50b8a4cca9d8f18159be80ff6c201ff7549169c1242a2abf029108972d" exitCode=0
Oct 30 21:55:50.870908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:50.870888    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6" event={"ID":"f5bf3460-e153-45ef-86db-837af9418a7d","Type":"ContainerDied","Data":"e0218e50b8a4cca9d8f18159be80ff6c201ff7549169c1242a2abf029108972d"}
Oct 30 21:55:50.873257 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:50.873232    4918 generic.go:356] "Generic (PLEG): container finished" podID="2c8d84c6-9d80-432e-bdc4-8e3684a473f8" containerID="278df32917c4ad613c26799354b6f5f77982a37833ed555fd21023f3391b253f" exitCode=0
Oct 30 21:55:50.873458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:50.873436    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd" event={"ID":"2c8d84c6-9d80-432e-bdc4-8e3684a473f8","Type":"ContainerDied","Data":"278df32917c4ad613c26799354b6f5f77982a37833ed555fd21023f3391b253f"}
Oct 30 21:55:50.874529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:50.874470    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px" podUID="d5cc4d1c-e96c-4df3-8b99-99257ffaf828" containerName="payload-pre-processing" containerID="cri-o://713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1" gracePeriod=25
Oct 30 21:55:51.070721 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.070694    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"
Oct 30 21:55:51.114628 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.114605    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"
Oct 30 21:55:51.215325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.215284    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-plugins-config-volume\") pod \"2c8d84c6-9d80-432e-bdc4-8e3684a473f8\" (UID: \"2c8d84c6-9d80-432e-bdc4-8e3684a473f8\") "
Oct 30 21:55:51.215843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.215343    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5bf3460-e153-45ef-86db-837af9418a7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bf3460-e153-45ef-86db-837af9418a7d-plugins-config-volume\") pod \"f5bf3460-e153-45ef-86db-837af9418a7d\" (UID: \"f5bf3460-e153-45ef-86db-837af9418a7d\") "
Oct 30 21:55:51.215843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.215413    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-kube-api-access-5p4vl\" (UniqueName: \"kubernetes.io/projected/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-kube-api-access-5p4vl\") pod \"2c8d84c6-9d80-432e-bdc4-8e3684a473f8\" (UID: \"2c8d84c6-9d80-432e-bdc4-8e3684a473f8\") "
Oct 30 21:55:51.215843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.215476    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5bf3460-e153-45ef-86db-837af9418a7d-kube-api-access-kf26b\" (UniqueName: \"kubernetes.io/projected/f5bf3460-e153-45ef-86db-837af9418a7d-kube-api-access-kf26b\") pod \"f5bf3460-e153-45ef-86db-837af9418a7d\" (UID: \"f5bf3460-e153-45ef-86db-837af9418a7d\") "
Oct 30 21:55:51.215843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.215770    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-plugins-config-volume" pod "2c8d84c6-9d80-432e-bdc4-8e3684a473f8" (UID: "2c8d84c6-9d80-432e-bdc4-8e3684a473f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:51.215843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.215801    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5bf3460-e153-45ef-86db-837af9418a7d-plugins-config-volume" pod "f5bf3460-e153-45ef-86db-837af9418a7d" (UID: "f5bf3460-e153-45ef-86db-837af9418a7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:51.216281 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.216256    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:51.216370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.216289    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5bf3460-e153-45ef-86db-837af9418a7d-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:51.218137 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.218101    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-kube-api-access-5p4vl" pod "2c8d84c6-9d80-432e-bdc4-8e3684a473f8" (UID: "2c8d84c6-9d80-432e-bdc4-8e3684a473f8"). InnerVolumeSpecName "kube-api-access-5p4vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:51.218244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.218144    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bf3460-e153-45ef-86db-837af9418a7d-kube-api-access-kf26b" pod "f5bf3460-e153-45ef-86db-837af9418a7d" (UID: "f5bf3460-e153-45ef-86db-837af9418a7d"). InnerVolumeSpecName "kube-api-access-kf26b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:51.317147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.317057    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p4vl\" (UniqueName: \"kubernetes.io/projected/2c8d84c6-9d80-432e-bdc4-8e3684a473f8-kube-api-access-5p4vl\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:51.317147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.317099    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf26b\" (UniqueName: \"kubernetes.io/projected/f5bf3460-e153-45ef-86db-837af9418a7d-kube-api-access-kf26b\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:51.855988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.855952    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"]
Oct 30 21:55:51.856729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856707    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ee87d020-7e1e-4f32-a2ab-3615215f49bf" containerName="revoke-keys"
Oct 30 21:55:51.856729 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856728    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee87d020-7e1e-4f32-a2ab-3615215f49bf" containerName="revoke-keys"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856753    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2c8d84c6-9d80-432e-bdc4-8e3684a473f8" containerName="payload-processing"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856759    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8d84c6-9d80-432e-bdc4-8e3684a473f8" containerName="payload-processing"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856772    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4db3211f-c998-428e-9364-88befce1cf71" containerName="payload-processing"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856777    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db3211f-c998-428e-9364-88befce1cf71" containerName="payload-processing"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856785    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f5bf3460-e153-45ef-86db-837af9418a7d" containerName="payload-pre-processing"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856793    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5bf3460-e153-45ef-86db-837af9418a7d" containerName="payload-pre-processing"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856806    4918 cpu_manager.go:414] "containerMap: removing container" podUID="755a5f2c-b99e-4c7b-893a-d346b9f92d81" containerName="payload-pre-processing"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856814    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755a5f2c-b99e-4c7b-893a-d346b9f92d81" containerName="payload-pre-processing"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856834    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" containerName="istio-proxy"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856842    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" containerName="istio-proxy"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856863    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerName="istio-proxy"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856870    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerName="istio-proxy"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856886    4918 cpu_manager.go:414] "containerMap: removing container" podUID="459a5db7-a125-41cb-91c0-8493b9748fd0" containerName="revoke-keys"
Oct 30 21:55:51.856890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.856893    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459a5db7-a125-41cb-91c0-8493b9748fd0" containerName="revoke-keys"
Oct 30 21:55:51.857929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.857054    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5bd77e8-a7c2-4bc2-b303-bf3ea8982c79" containerName="istio-proxy"
Oct 30 21:55:51.857929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.857069    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="459a5db7-a125-41cb-91c0-8493b9748fd0" containerName="revoke-keys"
Oct 30 21:55:51.857929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.857115    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee87d020-7e1e-4f32-a2ab-3615215f49bf" containerName="revoke-keys"
Oct 30 21:55:51.857929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.857129    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8d84c6-9d80-432e-bdc4-8e3684a473f8" containerName="payload-processing"
Oct 30 21:55:51.857929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.857138    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db3211f-c998-428e-9364-88befce1cf71" containerName="payload-processing"
Oct 30 21:55:51.857929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.857147    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5bf3460-e153-45ef-86db-837af9418a7d" containerName="payload-pre-processing"
Oct 30 21:55:51.857929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.857159    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="755a5f2c-b99e-4c7b-893a-d346b9f92d81" containerName="payload-pre-processing"
Oct 30 21:55:51.857929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.857170    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4378181-49fc-4f6b-b9ca-6700bec0fbc8" containerName="istio-proxy"
Oct 30 21:55:51.881365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.866916    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"]
Oct 30 21:55:51.881365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.867271    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:51.929431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.928791    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:55:51.937090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.934370    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"]
Oct 30 21:55:51.944118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.943439    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c61ec39f-k8wcz\" (UID: \"6b887953-b45a-4eb9-808c-dc5974efda1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:51.944118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.943522    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbmvz\" (UniqueName: \"kubernetes.io/projected/6b887953-b45a-4eb9-808c-dc5974efda1e-kube-api-access-jbmvz\") pod \"maas-api-revoke-sub-c61ec39f-k8wcz\" (UID: \"6b887953-b45a-4eb9-808c-dc5974efda1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:51.944118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.943564    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c61ec39f-k8wcz\" (UID: \"6b887953-b45a-4eb9-808c-dc5974efda1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:51.949963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.949940    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:51.989443 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:51.989377    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:55:52.046111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.046076    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"]
Oct 30 21:55:52.046497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.046163    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"]
Oct 30 21:55:52.055294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.053608    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"]
Oct 30 21:55:52.055294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.053821    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s" podUID="9d50305c-28b7-42d5-bda9-3dd6d0c13545" containerName="maas-api" containerID="cri-o://30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a" gracePeriod=30
Oct 30 21:55:52.055294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.053935    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c61ec39f-k8wcz\" (UID: \"6b887953-b45a-4eb9-808c-dc5974efda1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:52.055294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.054036    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c5a4a64-lvmch\" (UID: \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:52.055294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.054168    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5wt\" (UniqueName: \"kubernetes.io/projected/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-kube-api-access-kg5wt\") pod \"maas-api-revoke-sub-1c5a4a64-lvmch\" (UID: \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:52.055294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.054212    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c61ec39f-k8wcz\" (UID: \"6b887953-b45a-4eb9-808c-dc5974efda1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:52.055294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.054263    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbmvz\" (UniqueName: \"kubernetes.io/projected/6b887953-b45a-4eb9-808c-dc5974efda1e-kube-api-access-jbmvz\") pod \"maas-api-revoke-sub-c61ec39f-k8wcz\" (UID: \"6b887953-b45a-4eb9-808c-dc5974efda1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:52.055294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.054286    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c5a4a64-lvmch\" (UID: \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:52.055294 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.055147    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c61ec39f-k8wcz\" (UID: \"6b887953-b45a-4eb9-808c-dc5974efda1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:52.059681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.058739    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c61ec39f-k8wcz\" (UID: \"6b887953-b45a-4eb9-808c-dc5974efda1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:52.059681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.058822    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"
Oct 30 21:55:52.059681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.058924    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6" event={"ID":"f5bf3460-e153-45ef-86db-837af9418a7d","Type":"ContainerDied","Data":"ed529b5de69c70badc9372708cb10ac033f86673e470bc3d546a930b98ea1c9f"}
Oct 30 21:55:52.059681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.059005    4918 scope.go:122] "RemoveContainer" containerID="e0218e50b8a4cca9d8f18159be80ff6c201ff7549169c1242a2abf029108972d"
Oct 30 21:55:52.068300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.066964    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd" event={"ID":"2c8d84c6-9d80-432e-bdc4-8e3684a473f8","Type":"ContainerDied","Data":"8d148dc50c0a99306e2cf507e8afd8d399e034b884508f786de5d1b54ddd889a"}
Oct 30 21:55:52.068300 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.067062    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"
Oct 30 21:55:52.082994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.082968    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbmvz\" (UniqueName: \"kubernetes.io/projected/6b887953-b45a-4eb9-808c-dc5974efda1e-kube-api-access-jbmvz\") pod \"maas-api-revoke-sub-c61ec39f-k8wcz\" (UID: \"6b887953-b45a-4eb9-808c-dc5974efda1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:52.087725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.087694    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"]
Oct 30 21:55:52.155155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.155122    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c5a4a64-lvmch\" (UID: \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:52.155332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.155295    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5wt\" (UniqueName: \"kubernetes.io/projected/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-kube-api-access-kg5wt\") pod \"maas-api-revoke-sub-1c5a4a64-lvmch\" (UID: \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:52.155533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.155512    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c5a4a64-lvmch\" (UID: \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:52.156265 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.156238    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c5a4a64-lvmch\" (UID: \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:52.157707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.157688    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c5a4a64-lvmch\" (UID: \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:52.179546 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.179515    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5wt\" (UniqueName: \"kubernetes.io/projected/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-kube-api-access-kg5wt\") pod \"maas-api-revoke-sub-1c5a4a64-lvmch\" (UID: \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:52.188578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.188551    4918 scope.go:122] "RemoveContainer" containerID="278df32917c4ad613c26799354b6f5f77982a37833ed555fd21023f3391b253f"
Oct 30 21:55:52.191196 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.191140    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"]
Oct 30 21:55:52.220580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.195596    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-904993-wkrxd"]
Oct 30 21:55:52.221798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.221605    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:52.224139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.224106    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd" podUID="893a5d07-10fd-4b2f-a4a0-db937338f5db" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:52.230425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.227767    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"]
Oct 30 21:55:52.232802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.232746    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-d4a0a5-75456d66f9-rk4q6"]
Oct 30 21:55:52.254122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.253316    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"]
Oct 30 21:55:52.254122 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.253963    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d4a0a5-67965c4b8f-b52fd"]
Oct 30 21:55:52.374408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.372913    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:52.475675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.475651    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:55:52.554159 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:55:52.554134    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f9f82a_e844_44ff_8bb9_87cbaabb8dc3.slice/crio-f08bf7c206fb9bd7e9a615a83f876f2558c20e7585531ed5dca03debdbf8061e WatchSource:0}: Error finding container f08bf7c206fb9bd7e9a615a83f876f2558c20e7585531ed5dca03debdbf8061e: Status 404 returned error can't find the container with id f08bf7c206fb9bd7e9a615a83f876f2558c20e7585531ed5dca03debdbf8061e
Oct 30 21:55:52.557654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.557626    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"]
Oct 30 21:55:52.577225 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.577203    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d50305c-28b7-42d5-bda9-3dd6d0c13545-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d50305c-28b7-42d5-bda9-3dd6d0c13545-openshift-service-ca\") pod \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") "
Oct 30 21:55:52.577352 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.577274    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-maas-api-tls\") pod \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") "
Oct 30 21:55:52.577462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.577363    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d50305c-28b7-42d5-bda9-3dd6d0c13545-kube-api-access-zph4w\" (UniqueName: \"kubernetes.io/projected/9d50305c-28b7-42d5-bda9-3dd6d0c13545-kube-api-access-zph4w\") pod \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") "
Oct 30 21:55:52.577532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.577517    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-metrics-certs\") pod \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\" (UID: \"9d50305c-28b7-42d5-bda9-3dd6d0c13545\") "
Oct 30 21:55:52.577654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.577626    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d50305c-28b7-42d5-bda9-3dd6d0c13545-openshift-service-ca" pod "9d50305c-28b7-42d5-bda9-3dd6d0c13545" (UID: "9d50305c-28b7-42d5-bda9-3dd6d0c13545"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:52.578104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.578084    4918 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d50305c-28b7-42d5-bda9-3dd6d0c13545-openshift-service-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:52.580717 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.580636    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-maas-api-tls" pod "9d50305c-28b7-42d5-bda9-3dd6d0c13545" (UID: "9d50305c-28b7-42d5-bda9-3dd6d0c13545"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:52.581400 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.581302    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-metrics-certs" pod "9d50305c-28b7-42d5-bda9-3dd6d0c13545" (UID: "9d50305c-28b7-42d5-bda9-3dd6d0c13545"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:52.582792 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.582770    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d50305c-28b7-42d5-bda9-3dd6d0c13545-kube-api-access-zph4w" pod "9d50305c-28b7-42d5-bda9-3dd6d0c13545" (UID: "9d50305c-28b7-42d5-bda9-3dd6d0c13545"). InnerVolumeSpecName "kube-api-access-zph4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:52.640488 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.640452    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"]
Oct 30 21:55:52.643767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.643744    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e8d315-9c2lv"]
Oct 30 21:55:52.679545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.679520    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-maas-api-tls\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:52.679669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.679548    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zph4w\" (UniqueName: \"kubernetes.io/projected/9d50305c-28b7-42d5-bda9-3dd6d0c13545-kube-api-access-zph4w\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:52.679669 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.679563    4918 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d50305c-28b7-42d5-bda9-3dd6d0c13545-metrics-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:52.776943 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:52.776911    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"]
Oct 30 21:55:52.777102 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:55:52.777079    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b887953_b45a_4eb9_808c_dc5974efda1e.slice/crio-c1ea320d577f12896dd005623837b11256ecf1cb050bfd66aa8a5e0d30b16f27 WatchSource:0}: Error finding container c1ea320d577f12896dd005623837b11256ecf1cb050bfd66aa8a5e0d30b16f27: Status 404 returned error can't find the container with id c1ea320d577f12896dd005623837b11256ecf1cb050bfd66aa8a5e0d30b16f27
Oct 30 21:55:53.077681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.077600    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8d84c6-9d80-432e-bdc4-8e3684a473f8" path="/var/lib/kubelet/pods/2c8d84c6-9d80-432e-bdc4-8e3684a473f8/volumes"
Oct 30 21:55:53.078264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.078238    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60cbfccc-c9b7-4861-a9a2-7e156ffbaf68" path="/var/lib/kubelet/pods/60cbfccc-c9b7-4861-a9a2-7e156ffbaf68/volumes"
Oct 30 21:55:53.079412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.079367    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8b7f23a-6935-41cf-ad55-85f0b7cc4a42" path="/var/lib/kubelet/pods/e8b7f23a-6935-41cf-ad55-85f0b7cc4a42/volumes"
Oct 30 21:55:53.080141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.080118    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5bf3460-e153-45ef-86db-837af9418a7d" path="/var/lib/kubelet/pods/f5bf3460-e153-45ef-86db-837af9418a7d/volumes"
Oct 30 21:55:53.080945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.080917    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch" event={"ID":"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3","Type":"ContainerStarted","Data":"7d334b12d91d5ba4c8d422f757606c4c97ffa9b2a099777ceb01711036db9e93"}
Oct 30 21:55:53.081055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.080979    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch" event={"ID":"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3","Type":"ContainerStarted","Data":"f08bf7c206fb9bd7e9a615a83f876f2558c20e7585531ed5dca03debdbf8061e"}
Oct 30 21:55:53.103406 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.103363    4918 generic.go:356] "Generic (PLEG): container finished" podID="9d50305c-28b7-42d5-bda9-3dd6d0c13545" containerID="30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a" exitCode=0
Oct 30 21:55:53.103556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.103445    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"
Oct 30 21:55:53.103556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.103482    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s" event={"ID":"9d50305c-28b7-42d5-bda9-3dd6d0c13545","Type":"ContainerDied","Data":"30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a"}
Oct 30 21:55:53.103556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.103540    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s" event={"ID":"9d50305c-28b7-42d5-bda9-3dd6d0c13545","Type":"ContainerDied","Data":"0d0484a762837c979b38b55f97f39bf18a83da52ec14f78473ab1fe020743592"}
Oct 30 21:55:53.103723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.103569    4918 scope.go:122] "RemoveContainer" containerID="30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a"
Oct 30 21:55:53.104467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.104417    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch" podStartSLOduration=2.104383765 podStartE2EDuration="2.104383765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:53.10270902 +0000 UTC m=+3266.149268166" watchObservedRunningTime="2026-10-30 21:55:53.104383765 +0000 UTC m=+3266.150942911"
Oct 30 21:55:53.107098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.107077    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz" event={"ID":"6b887953-b45a-4eb9-808c-dc5974efda1e","Type":"ContainerStarted","Data":"c1ea320d577f12896dd005623837b11256ecf1cb050bfd66aa8a5e0d30b16f27"}
Oct 30 21:55:53.115297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.115277    4918 scope.go:122] "RemoveContainer" containerID="30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a"
Oct 30 21:55:53.115578 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:53.115561    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a\": container with ID starting with 30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a not found: ID does not exist" containerID="30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a"
Oct 30 21:55:53.115651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.115587    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a"} err="failed to get container status \"30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a\": rpc error: code = NotFound desc = could not find container \"30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a\": container with ID starting with 30511b79b580fbcca5b445cdd94bd024c8ea7c92f8193598a72b79f63dbb777a not found: ID does not exist"
Oct 30 21:55:53.127229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.127206    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"]
Oct 30 21:55:53.130498 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.130479    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e8d315-7ccbb9fd7-q9f5s"]
Oct 30 21:55:53.825335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.825287    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb" podUID="49552f5c-97a2-4efb-b7e7-674879eb98cf" containerName="payload-processing" containerID="cri-o://eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8" gracePeriod=25
Oct 30 21:55:53.838963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:53.838929    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs" podUID="c98af039-b120-4448-a7d9-a9e4a00ef226" containerName="payload-pre-processing" containerID="cri-o://166bc2dce5cb3cb357bd4566b40857caf66c6524f95beae268835119286a0be0" gracePeriod=25
Oct 30 21:55:54.117252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:54.117217    4918 generic.go:356] "Generic (PLEG): container finished" podID="6b887953-b45a-4eb9-808c-dc5974efda1e" containerID="7f4a6f85b50024103f0b97066d9f5d5f2f779cd0d45d9137a3dc0398af146cf4" exitCode=0
Oct 30 21:55:54.117576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:54.117282    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz" event={"ID":"6b887953-b45a-4eb9-808c-dc5974efda1e","Type":"ContainerDied","Data":"7f4a6f85b50024103f0b97066d9f5d5f2f779cd0d45d9137a3dc0398af146cf4"}
Oct 30 21:55:54.119009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:54.118982    4918 generic.go:356] "Generic (PLEG): container finished" podID="d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3" containerID="7d334b12d91d5ba4c8d422f757606c4c97ffa9b2a099777ceb01711036db9e93" exitCode=0
Oct 30 21:55:54.119135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:54.119056    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch" event={"ID":"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3","Type":"ContainerDied","Data":"7d334b12d91d5ba4c8d422f757606c4c97ffa9b2a099777ceb01711036db9e93"}
Oct 30 21:55:55.078409 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.078364    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d50305c-28b7-42d5-bda9-3dd6d0c13545" path="/var/lib/kubelet/pods/9d50305c-28b7-42d5-bda9-3dd6d0c13545/volumes"
Oct 30 21:55:55.338338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.338277    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:55.341698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.341677    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:55.416816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.416792    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-openshift-ca\") pod \"6b887953-b45a-4eb9-808c-dc5974efda1e\" (UID: \"6b887953-b45a-4eb9-808c-dc5974efda1e\") "
Oct 30 21:55:55.417019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.416864    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b887953-b45a-4eb9-808c-dc5974efda1e-kube-api-access-jbmvz\" (UniqueName: \"kubernetes.io/projected/6b887953-b45a-4eb9-808c-dc5974efda1e-kube-api-access-jbmvz\") pod \"6b887953-b45a-4eb9-808c-dc5974efda1e\" (UID: \"6b887953-b45a-4eb9-808c-dc5974efda1e\") "
Oct 30 21:55:55.417019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.416968    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-xks-ca\") pod \"6b887953-b45a-4eb9-808c-dc5974efda1e\" (UID: \"6b887953-b45a-4eb9-808c-dc5974efda1e\") "
Oct 30 21:55:55.417159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.417068    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-xks-ca\") pod \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\" (UID: \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\") "
Oct 30 21:55:55.417222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.417185    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-kube-api-access-kg5wt\" (UniqueName: \"kubernetes.io/projected/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-kube-api-access-kg5wt\") pod \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\" (UID: \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\") "
Oct 30 21:55:55.417323 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.417296    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-openshift-ca" pod "6b887953-b45a-4eb9-808c-dc5974efda1e" (UID: "6b887953-b45a-4eb9-808c-dc5974efda1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:55.417604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.417578    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-openshift-ca\") pod \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\" (UID: \"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3\") "
Oct 30 21:55:55.418601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.418575    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:55.421568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.420152    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-xks-ca" pod "d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3" (UID: "d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:55.421698 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.420480    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-openshift-ca" pod "d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3" (UID: "d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:55.421821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.420859    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-xks-ca" pod "6b887953-b45a-4eb9-808c-dc5974efda1e" (UID: "6b887953-b45a-4eb9-808c-dc5974efda1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:55.421821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.421134    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b887953-b45a-4eb9-808c-dc5974efda1e-kube-api-access-jbmvz" pod "6b887953-b45a-4eb9-808c-dc5974efda1e" (UID: "6b887953-b45a-4eb9-808c-dc5974efda1e"). InnerVolumeSpecName "kube-api-access-jbmvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:55.424124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.424096    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-kube-api-access-kg5wt" pod "d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3" (UID: "d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3"). InnerVolumeSpecName "kube-api-access-kg5wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:55.522585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.522557    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbmvz\" (UniqueName: \"kubernetes.io/projected/6b887953-b45a-4eb9-808c-dc5974efda1e-kube-api-access-jbmvz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:55.522585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.522585    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b887953-b45a-4eb9-808c-dc5974efda1e-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:55.522788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.522597    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:55.522788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.522606    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg5wt\" (UniqueName: \"kubernetes.io/projected/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-kube-api-access-kg5wt\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:55.522788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:55.522614    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:56.135074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:56.135045    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz"
Oct 30 21:55:56.135560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:56.135080    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61ec39f-k8wcz" event={"ID":"6b887953-b45a-4eb9-808c-dc5974efda1e","Type":"ContainerDied","Data":"c1ea320d577f12896dd005623837b11256ecf1cb050bfd66aa8a5e0d30b16f27"}
Oct 30 21:55:56.135560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:56.135135    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1ea320d577f12896dd005623837b11256ecf1cb050bfd66aa8a5e0d30b16f27"
Oct 30 21:55:56.136696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:56.136677    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch"
Oct 30 21:55:56.136812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:56.136748    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c5a4a64-lvmch" event={"ID":"d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3","Type":"ContainerDied","Data":"f08bf7c206fb9bd7e9a615a83f876f2558c20e7585531ed5dca03debdbf8061e"}
Oct 30 21:55:56.136812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:56.136782    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f08bf7c206fb9bd7e9a615a83f876f2558c20e7585531ed5dca03debdbf8061e"
Oct 30 21:55:56.681748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:56.681711    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px" podUID="d5cc4d1c-e96c-4df3-8b99-99257ffaf828" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:56.868034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:56.867993    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg" podUID="08b09d3f-a61b-4812-b128-c4ea5cea422f" containerName="payload-processing" containerID="cri-o://73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b" gracePeriod=25
Oct 30 21:55:57.047190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:57.047148    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9" podUID="791182e3-1f2f-400d-8577-fa86d6b74162" containerName="payload-pre-processing" containerID="cri-o://a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11" gracePeriod=25
Oct 30 21:55:57.124349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:57.124318    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"]
Oct 30 21:55:57.131181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:57.131157    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-27804d-nld6r"]
Oct 30 21:55:58.455810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.455767    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main" containerID="cri-o://1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145" gracePeriod=45
Oct 30 21:55:58.727163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.727140    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:55:58.772795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.772767    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-dshm\") pod \"d04b824b-1326-4aa0-885a-8997d849f347\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") "
Oct 30 21:55:58.772979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.772846    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04b824b-1326-4aa0-885a-8997d849f347-kube-api-access-r776b\" (UniqueName: \"kubernetes.io/projected/d04b824b-1326-4aa0-885a-8997d849f347-kube-api-access-r776b\") pod \"d04b824b-1326-4aa0-885a-8997d849f347\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") "
Oct 30 21:55:58.772979 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.772891    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-model-cache\") pod \"d04b824b-1326-4aa0-885a-8997d849f347\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") "
Oct 30 21:55:58.773119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.773003    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-home\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-home\") pod \"d04b824b-1326-4aa0-885a-8997d849f347\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") "
Oct 30 21:55:58.773119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.773055    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-tmp-dir\") pod \"d04b824b-1326-4aa0-885a-8997d849f347\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") "
Oct 30 21:55:58.773249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.773230    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d04b824b-1326-4aa0-885a-8997d849f347-tls-certs\" (UniqueName: \"kubernetes.io/secret/d04b824b-1326-4aa0-885a-8997d849f347-tls-certs\") pod \"d04b824b-1326-4aa0-885a-8997d849f347\" (UID: \"d04b824b-1326-4aa0-885a-8997d849f347\") "
Oct 30 21:55:58.773311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.773237    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-model-cache" pod "d04b824b-1326-4aa0-885a-8997d849f347" (UID: "d04b824b-1326-4aa0-885a-8997d849f347"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:58.773429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.773384    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-home" pod "d04b824b-1326-4aa0-885a-8997d849f347" (UID: "d04b824b-1326-4aa0-885a-8997d849f347"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:58.773825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.773788    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-tmp-dir" pod "d04b824b-1326-4aa0-885a-8997d849f347" (UID: "d04b824b-1326-4aa0-885a-8997d849f347"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:58.773948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.773880    4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:58.773948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.773903    4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:58.775367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.775340    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-dshm" pod "d04b824b-1326-4aa0-885a-8997d849f347" (UID: "d04b824b-1326-4aa0-885a-8997d849f347"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:55:58.775517 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.775470    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d04b824b-1326-4aa0-885a-8997d849f347-tls-certs" pod "d04b824b-1326-4aa0-885a-8997d849f347" (UID: "d04b824b-1326-4aa0-885a-8997d849f347"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:58.776252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.776223    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04b824b-1326-4aa0-885a-8997d849f347-kube-api-access-r776b" pod "d04b824b-1326-4aa0-885a-8997d849f347" (UID: "d04b824b-1326-4aa0-885a-8997d849f347"). InnerVolumeSpecName "kube-api-access-r776b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:58.875655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.875621    4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d04b824b-1326-4aa0-885a-8997d849f347-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:58.875655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.875652    4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:58.875655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.875661    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r776b\" (UniqueName: \"kubernetes.io/projected/d04b824b-1326-4aa0-885a-8997d849f347-kube-api-access-r776b\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:58.875974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:58.875672    4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d04b824b-1326-4aa0-885a-8997d849f347-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:55:59.080237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:59.080201    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee87d020-7e1e-4f32-a2ab-3615215f49bf" path="/var/lib/kubelet/pods/ee87d020-7e1e-4f32-a2ab-3615215f49bf/volumes"
Oct 30 21:55:59.160608 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:59.160579    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:55:59.161354 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:59.161328    4918 generic.go:356] "Generic (PLEG): container finished" podID="d04b824b-1326-4aa0-885a-8997d849f347" containerID="1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145" exitCode=0
Oct 30 21:55:59.161487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:59.161375    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"
Oct 30 21:55:59.161487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:59.161422    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" event={"ID":"d04b824b-1326-4aa0-885a-8997d849f347","Type":"ContainerDied","Data":"1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145"}
Oct 30 21:55:59.161487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:59.161474    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89" event={"ID":"d04b824b-1326-4aa0-885a-8997d849f347","Type":"ContainerDied","Data":"f52cc62c98fcfdd185b139442207154ffae88bf7254da81827081d07d250f359"}
Oct 30 21:55:59.161639 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:59.161491    4918 scope.go:122] "RemoveContainer" containerID="1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145"
Oct 30 21:55:59.172611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:59.172596    4918 scope.go:122] "RemoveContainer" containerID="1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145"
Oct 30 21:55:59.172854 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:55:59.172836    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145\": container with ID starting with 1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145 not found: ID does not exist" containerID="1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145"
Oct 30 21:55:59.172906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:59.172870    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145"} err="failed to get container status \"1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145\": rpc error: code = NotFound desc = could not find container \"1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145\": container with ID starting with 1b3a7c0a5bc168bb0502f0b41c570ee40509a6c864d50ee48792de2c15519145 not found: ID does not exist"
Oct 30 21:55:59.185431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:59.185379    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"]
Oct 30 21:55:59.192447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:59.192418    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bfc50e/simulator-w3-bfc50e-kserve-84df57b68c-fpr89"]
Oct 30 21:55:59.659248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:59.659205    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg" podUID="08b09d3f-a61b-4812-b128-c4ea5cea422f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:55:59.659248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:55:59.659232    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9" podUID="791182e3-1f2f-400d-8577-fa86d6b74162" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:00.030072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:00.030038    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs" podUID="c98af039-b120-4448-a7d9-a9e4a00ef226" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:00.030072 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:00.030053    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb" podUID="49552f5c-97a2-4efb-b7e7-674879eb98cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:00.168307 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:00.168270    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg" podUID="cc7098ea-8a7e-4286-89e2-c3f6bf14d92b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:01.079101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:01.079070    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d04b824b-1326-4aa0-885a-8997d849f347" path="/var/lib/kubelet/pods/d04b824b-1326-4aa0-885a-8997d849f347/volumes"
Oct 30 21:56:01.483892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:01.483863    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"]
Oct 30 21:56:01.484092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:01.484071    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy" containerID="cri-o://09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0" gracePeriod=30
Oct 30 21:56:02.171570 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.171540    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"]
Oct 30 21:56:02.172112 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.172097    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3" containerName="revoke-keys"
Oct 30 21:56:02.172161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.172113    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3" containerName="revoke-keys"
Oct 30 21:56:02.172161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.172127    4918 cpu_manager.go:414] "containerMap: removing container" podUID="6b887953-b45a-4eb9-808c-dc5974efda1e" containerName="revoke-keys"
Oct 30 21:56:02.172161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.172132    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b887953-b45a-4eb9-808c-dc5974efda1e" containerName="revoke-keys"
Oct 30 21:56:02.172161 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.172157    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main"
Oct 30 21:56:02.172287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.172163    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main"
Oct 30 21:56:02.172287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.172176    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9d50305c-28b7-42d5-bda9-3dd6d0c13545" containerName="maas-api"
Oct 30 21:56:02.172287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.172181    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d50305c-28b7-42d5-bda9-3dd6d0c13545" containerName="maas-api"
Oct 30 21:56:02.172287 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.172282    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04b824b-1326-4aa0-885a-8997d849f347" containerName="main"
Oct 30 21:56:02.172442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.172292    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d50305c-28b7-42d5-bda9-3dd6d0c13545" containerName="maas-api"
Oct 30 21:56:02.172442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.172302    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3" containerName="revoke-keys"
Oct 30 21:56:02.172442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.172308    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b887953-b45a-4eb9-808c-dc5974efda1e" containerName="revoke-keys"
Oct 30 21:56:02.177235 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.177221    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:02.189864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.189071    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:56:02.189864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.189063    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:56:02.190286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.190269    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:02.190455 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.190434    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:56:02.193370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.193343    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"]
Oct 30 21:56:02.217625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.217594    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd" podUID="893a5d07-10fd-4b2f-a4a0-db937338f5db" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:02.326260 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.326221    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9\" (UID: \"c84e9383-450d-49af-acf2-aa4ff741af9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:02.326447 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.326271    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9\" (UID: \"c84e9383-450d-49af-acf2-aa4ff741af9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:02.326521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.326497    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq55n\" (UniqueName: \"kubernetes.io/projected/c84e9383-450d-49af-acf2-aa4ff741af9e-kube-api-access-cq55n\") pod \"maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9\" (UID: \"c84e9383-450d-49af-acf2-aa4ff741af9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:02.428249 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.428163    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9\" (UID: \"c84e9383-450d-49af-acf2-aa4ff741af9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:02.428527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.428340    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq55n\" (UniqueName: \"kubernetes.io/projected/c84e9383-450d-49af-acf2-aa4ff741af9e-kube-api-access-cq55n\") pod \"maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9\" (UID: \"c84e9383-450d-49af-acf2-aa4ff741af9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:02.428527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.428432    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9\" (UID: \"c84e9383-450d-49af-acf2-aa4ff741af9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:02.429011 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.428989    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9\" (UID: \"c84e9383-450d-49af-acf2-aa4ff741af9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:02.430670 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.430649    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9\" (UID: \"c84e9383-450d-49af-acf2-aa4ff741af9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:02.439073 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.439053    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq55n\" (UniqueName: \"kubernetes.io/projected/c84e9383-450d-49af-acf2-aa4ff741af9e-kube-api-access-cq55n\") pod \"maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9\" (UID: \"c84e9383-450d-49af-acf2-aa4ff741af9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:02.491814 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.491793    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:02.805685 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:56:02.805651    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84e9383_450d_49af_acf2_aa4ff741af9e.slice/crio-7682fa931f642b10ed45f5808fec5b0000d169f0b03825ef14045018de590a34 WatchSource:0}: Error finding container 7682fa931f642b10ed45f5808fec5b0000d169f0b03825ef14045018de590a34: Status 404 returned error can't find the container with id 7682fa931f642b10ed45f5808fec5b0000d169f0b03825ef14045018de590a34
Oct 30 21:56:02.813491 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:02.813465    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"]
Oct 30 21:56:03.120298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.120262    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"]
Oct 30 21:56:03.120560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.120531    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv" podUID="a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" containerName="istio-proxy" containerID="cri-o://1bc4255c69d7772fd341b751783226ef3758b6bcc2a8bfe1b517380ac23bba61" gracePeriod=30
Oct 30 21:56:03.208724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.208695    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9" event={"ID":"c84e9383-450d-49af-acf2-aa4ff741af9e","Type":"ContainerStarted","Data":"f4dc6c31a3bc189820405f6ea0010299db9541cb935bd6e41e1dc483af8c8b26"}
Oct 30 21:56:03.209135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.208753    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9" event={"ID":"c84e9383-450d-49af-acf2-aa4ff741af9e","Type":"ContainerStarted","Data":"7682fa931f642b10ed45f5808fec5b0000d169f0b03825ef14045018de590a34"}
Oct 30 21:56:03.263877 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.263809    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9" podStartSLOduration=1.263788221 podStartE2EDuration="1.263788221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:03.259988231 +0000 UTC m=+3276.306547389" watchObservedRunningTime="2026-10-30 21:56:03.263788221 +0000 UTC m=+3276.310347367"
Oct 30 21:56:03.618275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.618236    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"]
Oct 30 21:56:03.622463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.622441    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:03.640825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.640794    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"]
Oct 30 21:56:03.746868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.746841    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn\" (UID: \"43d12812-6aa1-423f-acee-7ac62c91b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:03.746986 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.746889    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjzj\" (UniqueName: \"kubernetes.io/projected/43d12812-6aa1-423f-acee-7ac62c91b0c3-kube-api-access-2jjzj\") pod \"maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn\" (UID: \"43d12812-6aa1-423f-acee-7ac62c91b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:03.747042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.747017    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn\" (UID: \"43d12812-6aa1-423f-acee-7ac62c91b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:03.848023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.847991    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn\" (UID: \"43d12812-6aa1-423f-acee-7ac62c91b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:03.848195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.848043    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjzj\" (UniqueName: \"kubernetes.io/projected/43d12812-6aa1-423f-acee-7ac62c91b0c3-kube-api-access-2jjzj\") pod \"maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn\" (UID: \"43d12812-6aa1-423f-acee-7ac62c91b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:03.848195 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.848124    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn\" (UID: \"43d12812-6aa1-423f-acee-7ac62c91b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:03.848785 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.848749    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn\" (UID: \"43d12812-6aa1-423f-acee-7ac62c91b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:03.850548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.850527    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn\" (UID: \"43d12812-6aa1-423f-acee-7ac62c91b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:03.871708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.871653    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjzj\" (UniqueName: \"kubernetes.io/projected/43d12812-6aa1-423f-acee-7ac62c91b0c3-kube-api-access-2jjzj\") pod \"maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn\" (UID: \"43d12812-6aa1-423f-acee-7ac62c91b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:03.973896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:03.973604    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:04.219529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:04.219452    4918 generic.go:356] "Generic (PLEG): container finished" podID="c84e9383-450d-49af-acf2-aa4ff741af9e" containerID="f4dc6c31a3bc189820405f6ea0010299db9541cb935bd6e41e1dc483af8c8b26" exitCode=0
Oct 30 21:56:04.219529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:04.219492    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9" event={"ID":"c84e9383-450d-49af-acf2-aa4ff741af9e","Type":"ContainerDied","Data":"f4dc6c31a3bc189820405f6ea0010299db9541cb935bd6e41e1dc483af8c8b26"}
Oct 30 21:56:04.323907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:04.323874    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"]
Oct 30 21:56:04.325252 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:56:04.325224    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d12812_6aa1_423f_acee_7ac62c91b0c3.slice/crio-725586e64cfd36af0b2303a23f13f31cd69c36ccc951202187bbe4d1ba2d3784 WatchSource:0}: Error finding container 725586e64cfd36af0b2303a23f13f31cd69c36ccc951202187bbe4d1ba2d3784: Status 404 returned error can't find the container with id 725586e64cfd36af0b2303a23f13f31cd69c36ccc951202187bbe4d1ba2d3784
Oct 30 21:56:05.241470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.241435    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn" event={"ID":"43d12812-6aa1-423f-acee-7ac62c91b0c3","Type":"ContainerStarted","Data":"01bd1e231590cbb50b26164a1713474764dd390d441c6f7e795fd972a3c15925"}
Oct 30 21:56:05.241987 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.241530    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn" event={"ID":"43d12812-6aa1-423f-acee-7ac62c91b0c3","Type":"ContainerStarted","Data":"725586e64cfd36af0b2303a23f13f31cd69c36ccc951202187bbe4d1ba2d3784"}
Oct 30 21:56:05.293370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.293316    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn" podStartSLOduration=2.293299058 podStartE2EDuration="2.293299058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:05.288518484 +0000 UTC m=+3278.335077655" watchObservedRunningTime="2026-10-30 21:56:05.293299058 +0000 UTC m=+3278.339858203"
Oct 30 21:56:05.392551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.392526    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:05.416027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.416001    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-xks-ca\") pod \"c84e9383-450d-49af-acf2-aa4ff741af9e\" (UID: \"c84e9383-450d-49af-acf2-aa4ff741af9e\") "
Oct 30 21:56:05.416174 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.416052    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84e9383-450d-49af-acf2-aa4ff741af9e-kube-api-access-cq55n\" (UniqueName: \"kubernetes.io/projected/c84e9383-450d-49af-acf2-aa4ff741af9e-kube-api-access-cq55n\") pod \"c84e9383-450d-49af-acf2-aa4ff741af9e\" (UID: \"c84e9383-450d-49af-acf2-aa4ff741af9e\") "
Oct 30 21:56:05.416442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.416281    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-openshift-ca\") pod \"c84e9383-450d-49af-acf2-aa4ff741af9e\" (UID: \"c84e9383-450d-49af-acf2-aa4ff741af9e\") "
Oct 30 21:56:05.416644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.416621    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-openshift-ca" pod "c84e9383-450d-49af-acf2-aa4ff741af9e" (UID: "c84e9383-450d-49af-acf2-aa4ff741af9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:05.418180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.418157    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-xks-ca" pod "c84e9383-450d-49af-acf2-aa4ff741af9e" (UID: "c84e9383-450d-49af-acf2-aa4ff741af9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:05.418275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.418254    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84e9383-450d-49af-acf2-aa4ff741af9e-kube-api-access-cq55n" pod "c84e9383-450d-49af-acf2-aa4ff741af9e" (UID: "c84e9383-450d-49af-acf2-aa4ff741af9e"). InnerVolumeSpecName "kube-api-access-cq55n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:05.518343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.518258    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:05.518343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.518292    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84e9383-450d-49af-acf2-aa4ff741af9e-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:05.518343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:05.518305    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq55n\" (UniqueName: \"kubernetes.io/projected/c84e9383-450d-49af-acf2-aa4ff741af9e-kube-api-access-cq55n\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:06.250880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.250851    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"
Oct 30 21:56:06.251370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.250888    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9" event={"ID":"c84e9383-450d-49af-acf2-aa4ff741af9e","Type":"ContainerDied","Data":"7682fa931f642b10ed45f5808fec5b0000d169f0b03825ef14045018de590a34"}
Oct 30 21:56:06.251370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.250938    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7682fa931f642b10ed45f5808fec5b0000d169f0b03825ef14045018de590a34"
Oct 30 21:56:06.252873 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.252843    4918 generic.go:356] "Generic (PLEG): container finished" podID="43d12812-6aa1-423f-acee-7ac62c91b0c3" containerID="01bd1e231590cbb50b26164a1713474764dd390d441c6f7e795fd972a3c15925" exitCode=0
Oct 30 21:56:06.253010 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.252892    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn" event={"ID":"43d12812-6aa1-423f-acee-7ac62c91b0c3","Type":"ContainerDied","Data":"01bd1e231590cbb50b26164a1713474764dd390d441c6f7e795fd972a3c15925"}
Oct 30 21:56:06.682251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.682217    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px" podUID="d5cc4d1c-e96c-4df3-8b99-99257ffaf828" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:06.766036 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.765980    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:56:06.846337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.846308    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-podinfo\") pod \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") "
Oct 30 21:56:06.846537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.846346    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-envoy\") pod \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") "
Oct 30 21:56:06.846537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.846399    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-certs\") pod \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") "
Oct 30 21:56:06.846537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.846502    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-data\") pod \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") "
Oct 30 21:56:06.846537 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.846532    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-kube-api-access-glz9q\" (UniqueName: \"kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-kube-api-access-glz9q\") pod \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") "
Oct 30 21:56:06.846789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.846550    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-socket\") pod \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") "
Oct 30 21:56:06.846789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.846583    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istiod-ca-cert\") pod \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") "
Oct 30 21:56:06.846789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.846615    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-token\" (UniqueName: \"kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-token\") pod \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") "
Oct 30 21:56:06.846789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.846667    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-credential-socket\") pod \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\" (UID: \"da7e7720-3b01-4e3b-bdfe-5c771d4346aa\") "
Oct 30 21:56:06.846989 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.846764    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-certs" pod "da7e7720-3b01-4e3b-bdfe-5c771d4346aa" (UID: "da7e7720-3b01-4e3b-bdfe-5c771d4346aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:06.847056 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.846970    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-data" pod "da7e7720-3b01-4e3b-bdfe-5c771d4346aa" (UID: "da7e7720-3b01-4e3b-bdfe-5c771d4346aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:06.847120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.847067    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-credential-socket" pod "da7e7720-3b01-4e3b-bdfe-5c771d4346aa" (UID: "da7e7720-3b01-4e3b-bdfe-5c771d4346aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:06.847233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.847172    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istiod-ca-cert" pod "da7e7720-3b01-4e3b-bdfe-5c771d4346aa" (UID: "da7e7720-3b01-4e3b-bdfe-5c771d4346aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:06.847233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.847025    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-socket" pod "da7e7720-3b01-4e3b-bdfe-5c771d4346aa" (UID: "da7e7720-3b01-4e3b-bdfe-5c771d4346aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:06.847480 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.847459    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:06.847538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.847487    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:06.847538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.847504    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:06.849114 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.849071    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-kube-api-access-glz9q" pod "da7e7720-3b01-4e3b-bdfe-5c771d4346aa" (UID: "da7e7720-3b01-4e3b-bdfe-5c771d4346aa"). InnerVolumeSpecName "kube-api-access-glz9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:06.849229 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.849209    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-token" pod "da7e7720-3b01-4e3b-bdfe-5c771d4346aa" (UID: "da7e7720-3b01-4e3b-bdfe-5c771d4346aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:06.849343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.849330    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-podinfo" pod "da7e7720-3b01-4e3b-bdfe-5c771d4346aa" (UID: "da7e7720-3b01-4e3b-bdfe-5c771d4346aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:06.849434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.849364    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-envoy" pod "da7e7720-3b01-4e3b-bdfe-5c771d4346aa" (UID: "da7e7720-3b01-4e3b-bdfe-5c771d4346aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:06.949045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.949010    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:06.949045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.949041    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:06.949045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.949050    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glz9q\" (UniqueName: \"kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-kube-api-access-glz9q\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:06.949279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.949061    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:06.949279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.949070    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:06.949279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:06.949079    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da7e7720-3b01-4e3b-bdfe-5c771d4346aa-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:07.063825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.063718    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"]
Oct 30 21:56:07.081783 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.081751    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"]
Oct 30 21:56:07.084301 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.084277    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"]
Oct 30 21:56:07.262525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.262480    4918 generic.go:356] "Generic (PLEG): container finished" podID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerID="09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0" exitCode=0
Oct 30 21:56:07.262525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.262522    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" event={"ID":"da7e7720-3b01-4e3b-bdfe-5c771d4346aa","Type":"ContainerDied","Data":"09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0"}
Oct 30 21:56:07.262525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.262547    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"
Oct 30 21:56:07.263426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.262598    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm" event={"ID":"da7e7720-3b01-4e3b-bdfe-5c771d4346aa","Type":"ContainerDied","Data":"37d9eeed0dba582d03916eb44b0fdd0e82bed8bd8067636a435dfc3359b34786"}
Oct 30 21:56:07.263426 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.262636    4918 scope.go:122] "RemoveContainer" containerID="09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0"
Oct 30 21:56:07.275439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.275418    4918 scope.go:122] "RemoveContainer" containerID="c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085"
Oct 30 21:56:07.286530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.286501    4918 scope.go:122] "RemoveContainer" containerID="09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0"
Oct 30 21:56:07.286869 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:07.286847    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0\": container with ID starting with 09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0 not found: ID does not exist" containerID="09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0"
Oct 30 21:56:07.286950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.286891    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0"} err="failed to get container status \"09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0\": rpc error: code = NotFound desc = could not find container \"09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0\": container with ID starting with 09fe5da7dc2017b5bf0d368d410e59311db625f84220a0b874dae2f4258e7ea0 not found: ID does not exist"
Oct 30 21:56:07.286950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.286910    4918 scope.go:122] "RemoveContainer" containerID="c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085"
Oct 30 21:56:07.287197 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:07.287177    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085\": container with ID starting with c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085 not found: ID does not exist" containerID="c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085"
Oct 30 21:56:07.287257 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.287205    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085"} err="failed to get container status \"c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085\": rpc error: code = NotFound desc = could not find container \"c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085\": container with ID starting with c524c886eabef98e36ac72981137a01b9b05e82c9a2516cc46c768459cc41085 not found: ID does not exist"
Oct 30 21:56:07.341201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.341169    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"]
Oct 30 21:56:07.350661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.350633    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-904993-openshift-default-684b94bd69-gfthm"]
Oct 30 21:56:07.489213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.489189    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:07.557115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.557076    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-xks-ca\") pod \"43d12812-6aa1-423f-acee-7ac62c91b0c3\" (UID: \"43d12812-6aa1-423f-acee-7ac62c91b0c3\") "
Oct 30 21:56:07.557289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.557227    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d12812-6aa1-423f-acee-7ac62c91b0c3-kube-api-access-2jjzj\" (UniqueName: \"kubernetes.io/projected/43d12812-6aa1-423f-acee-7ac62c91b0c3-kube-api-access-2jjzj\") pod \"43d12812-6aa1-423f-acee-7ac62c91b0c3\" (UID: \"43d12812-6aa1-423f-acee-7ac62c91b0c3\") "
Oct 30 21:56:07.557367 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.557304    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-openshift-ca\") pod \"43d12812-6aa1-423f-acee-7ac62c91b0c3\" (UID: \"43d12812-6aa1-423f-acee-7ac62c91b0c3\") "
Oct 30 21:56:07.558754 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.558721    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-openshift-ca" pod "43d12812-6aa1-423f-acee-7ac62c91b0c3" (UID: "43d12812-6aa1-423f-acee-7ac62c91b0c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:07.570801 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.570753    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d12812-6aa1-423f-acee-7ac62c91b0c3-kube-api-access-2jjzj" pod "43d12812-6aa1-423f-acee-7ac62c91b0c3" (UID: "43d12812-6aa1-423f-acee-7ac62c91b0c3"). InnerVolumeSpecName "kube-api-access-2jjzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:07.572436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.571892    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-xks-ca" pod "43d12812-6aa1-423f-acee-7ac62c91b0c3" (UID: "43d12812-6aa1-423f-acee-7ac62c91b0c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:07.655541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.655509    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"
Oct 30 21:56:07.658974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.658952    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jjzj\" (UniqueName: \"kubernetes.io/projected/43d12812-6aa1-423f-acee-7ac62c91b0c3-kube-api-access-2jjzj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:07.659041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.658978    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:07.659041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.658988    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d12812-6aa1-423f-acee-7ac62c91b0c3-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:07.686109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.686082    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"
Oct 30 21:56:07.760137 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.760110    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-plugins-config-volume\") pod \"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b\" (UID: \"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b\") "
Oct 30 21:56:07.760322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.760246    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-kube-api-access-g6ds2\" (UniqueName: \"kubernetes.io/projected/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-kube-api-access-g6ds2\") pod \"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b\" (UID: \"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b\") "
Oct 30 21:56:07.760322 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.760303    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893a5d07-10fd-4b2f-a4a0-db937338f5db-kube-api-access-7bnsp\" (UniqueName: \"kubernetes.io/projected/893a5d07-10fd-4b2f-a4a0-db937338f5db-kube-api-access-7bnsp\") pod \"893a5d07-10fd-4b2f-a4a0-db937338f5db\" (UID: \"893a5d07-10fd-4b2f-a4a0-db937338f5db\") "
Oct 30 21:56:07.760490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.760341    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893a5d07-10fd-4b2f-a4a0-db937338f5db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893a5d07-10fd-4b2f-a4a0-db937338f5db-plugins-config-volume\") pod \"893a5d07-10fd-4b2f-a4a0-db937338f5db\" (UID: \"893a5d07-10fd-4b2f-a4a0-db937338f5db\") "
Oct 30 21:56:07.760738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.760570    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-plugins-config-volume" pod "cc7098ea-8a7e-4286-89e2-c3f6bf14d92b" (UID: "cc7098ea-8a7e-4286-89e2-c3f6bf14d92b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:07.760881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.760755    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893a5d07-10fd-4b2f-a4a0-db937338f5db-plugins-config-volume" pod "893a5d07-10fd-4b2f-a4a0-db937338f5db" (UID: "893a5d07-10fd-4b2f-a4a0-db937338f5db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:07.761070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.761048    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/893a5d07-10fd-4b2f-a4a0-db937338f5db-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:07.761163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.761075    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:07.762903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.762869    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-kube-api-access-g6ds2" pod "cc7098ea-8a7e-4286-89e2-c3f6bf14d92b" (UID: "cc7098ea-8a7e-4286-89e2-c3f6bf14d92b"). InnerVolumeSpecName "kube-api-access-g6ds2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:07.763262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.763242    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893a5d07-10fd-4b2f-a4a0-db937338f5db-kube-api-access-7bnsp" pod "893a5d07-10fd-4b2f-a4a0-db937338f5db" (UID: "893a5d07-10fd-4b2f-a4a0-db937338f5db"). InnerVolumeSpecName "kube-api-access-7bnsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:07.862528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.862451    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bnsp\" (UniqueName: \"kubernetes.io/projected/893a5d07-10fd-4b2f-a4a0-db937338f5db-kube-api-access-7bnsp\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:07.862528 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:07.862480    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6ds2\" (UniqueName: \"kubernetes.io/projected/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b-kube-api-access-g6ds2\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.274463 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.274428    4918 generic.go:356] "Generic (PLEG): container finished" podID="cc7098ea-8a7e-4286-89e2-c3f6bf14d92b" containerID="76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578" exitCode=0
Oct 30 21:56:08.274908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.274480    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"
Oct 30 21:56:08.274908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.274539    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg" event={"ID":"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b","Type":"ContainerDied","Data":"76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578"}
Oct 30 21:56:08.274908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.274586    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg" event={"ID":"cc7098ea-8a7e-4286-89e2-c3f6bf14d92b","Type":"ContainerDied","Data":"aac92a4a84354805c16b1e1d93b6b298b932eda2f2feb5a3e5e2dd18b43cfc1f"}
Oct 30 21:56:08.274908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.274607    4918 scope.go:122] "RemoveContainer" containerID="76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578"
Oct 30 21:56:08.276697 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.276671    4918 generic.go:356] "Generic (PLEG): container finished" podID="a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" containerID="1bc4255c69d7772fd341b751783226ef3758b6bcc2a8bfe1b517380ac23bba61" exitCode=0
Oct 30 21:56:08.276847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.276784    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv" event={"ID":"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba","Type":"ContainerDied","Data":"1bc4255c69d7772fd341b751783226ef3758b6bcc2a8bfe1b517380ac23bba61"}
Oct 30 21:56:08.278464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.278435    4918 generic.go:356] "Generic (PLEG): container finished" podID="893a5d07-10fd-4b2f-a4a0-db937338f5db" containerID="ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136" exitCode=0
Oct 30 21:56:08.278578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.278507    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd" event={"ID":"893a5d07-10fd-4b2f-a4a0-db937338f5db","Type":"ContainerDied","Data":"ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136"}
Oct 30 21:56:08.278578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.278551    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd" event={"ID":"893a5d07-10fd-4b2f-a4a0-db937338f5db","Type":"ContainerDied","Data":"3487086a967f5e36dd4e125cb05bd86b538082819423a62450c1f6083229aa5c"}
Oct 30 21:56:08.278578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.278491    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"
Oct 30 21:56:08.280441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.280416    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn" event={"ID":"43d12812-6aa1-423f-acee-7ac62c91b0c3","Type":"ContainerDied","Data":"725586e64cfd36af0b2303a23f13f31cd69c36ccc951202187bbe4d1ba2d3784"}
Oct 30 21:56:08.280561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.280462    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="725586e64cfd36af0b2303a23f13f31cd69c36ccc951202187bbe4d1ba2d3784"
Oct 30 21:56:08.280561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.280520    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"
Oct 30 21:56:08.292312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.292295    4918 scope.go:122] "RemoveContainer" containerID="76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578"
Oct 30 21:56:08.292648 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:08.292629    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578\": container with ID starting with 76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578 not found: ID does not exist" containerID="76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578"
Oct 30 21:56:08.292747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.292656    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578"} err="failed to get container status \"76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578\": rpc error: code = NotFound desc = could not find container \"76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578\": container with ID starting with 76a20c6b1042b3aa29a6ff1f530a02ddb855c12d420380bd667522d81979e578 not found: ID does not exist"
Oct 30 21:56:08.292747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.292678    4918 scope.go:122] "RemoveContainer" containerID="ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136"
Oct 30 21:56:08.305252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.305231    4918 scope.go:122] "RemoveContainer" containerID="ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136"
Oct 30 21:56:08.305658 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:08.305629    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136\": container with ID starting with ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136 not found: ID does not exist" containerID="ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136"
Oct 30 21:56:08.305767 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.305669    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136"} err="failed to get container status \"ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136\": rpc error: code = NotFound desc = could not find container \"ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136\": container with ID starting with ad42c2917e5ca09a2ad8e8b1bee3cfeb2e688e4cc6c9a90d71cbb3b9f8c2b136 not found: ID does not exist"
Oct 30 21:56:08.312087 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.312051    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"]
Oct 30 21:56:08.319810 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.319785    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-904993-7dcd9477b7-2wsqg"]
Oct 30 21:56:08.354095 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.354058    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"]
Oct 30 21:56:08.361133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.361106    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-904993-5865897dd7-vxngd"]
Oct 30 21:56:08.390572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.390521    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:56:08.469151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469122    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-token\" (UniqueName: \"kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-token\") pod \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") "
Oct 30 21:56:08.469304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469166    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-socket\") pod \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") "
Oct 30 21:56:08.469304 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469202    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-data\") pod \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") "
Oct 30 21:56:08.469433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469351    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-credential-socket\") pod \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") "
Oct 30 21:56:08.469493 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469467    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-data" pod "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" (UID: "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:08.469563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469487    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-certs\") pod \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") "
Oct 30 21:56:08.469563 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469527    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-kube-api-access-bqcz2\" (UniqueName: \"kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-kube-api-access-bqcz2\") pod \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") "
Oct 30 21:56:08.469683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469561    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-socket" pod "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" (UID: "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:08.469683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469597    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-podinfo\") pod \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") "
Oct 30 21:56:08.469816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469703    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-credential-socket" pod "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" (UID: "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:08.469816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469706    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istiod-ca-cert\") pod \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") "
Oct 30 21:56:08.469816 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469733    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-certs" pod "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" (UID: "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:08.470003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.469820    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-envoy\") pod \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\" (UID: \"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba\") "
Oct 30 21:56:08.470344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.470278    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istiod-ca-cert" pod "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" (UID: "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:08.470633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.470453    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.470633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.470474    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.470633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.470489    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.470633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.470504    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.470633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.470517    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.472011 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.471988    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-token" pod "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" (UID: "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:08.472118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.472083    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-kube-api-access-bqcz2" pod "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" (UID: "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba"). InnerVolumeSpecName "kube-api-access-bqcz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:08.472365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.472344    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-podinfo" pod "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" (UID: "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:08.472465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.472447    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-envoy" pod "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" (UID: "a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:08.571693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.571662    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.571693 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.571691    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.571914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.571703    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqcz2\" (UniqueName: \"kubernetes.io/projected/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-kube-api-access-bqcz2\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:08.571914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:08.571713    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:09.077999 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:09.077967    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893a5d07-10fd-4b2f-a4a0-db937338f5db" path="/var/lib/kubelet/pods/893a5d07-10fd-4b2f-a4a0-db937338f5db/volumes"
Oct 30 21:56:09.078425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:09.078410    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc7098ea-8a7e-4286-89e2-c3f6bf14d92b" path="/var/lib/kubelet/pods/cc7098ea-8a7e-4286-89e2-c3f6bf14d92b/volumes"
Oct 30 21:56:09.078861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:09.078848    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" path="/var/lib/kubelet/pods/da7e7720-3b01-4e3b-bdfe-5c771d4346aa/volumes"
Oct 30 21:56:09.291014 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:09.290984    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"
Oct 30 21:56:09.291538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:09.291019    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv" event={"ID":"a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba","Type":"ContainerDied","Data":"919744ed21eee41eac609c0a586240f0014c8880deef7a3e07bd63049c36bfb1"}
Oct 30 21:56:09.291538 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:09.291082    4918 scope.go:122] "RemoveContainer" containerID="1bc4255c69d7772fd341b751783226ef3758b6bcc2a8bfe1b517380ac23bba61"
Oct 30 21:56:09.316086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:09.316060    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"]
Oct 30 21:56:09.321013 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:09.320992    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d315-openshift-default-56b45c6865-ktfzv"]
Oct 30 21:56:09.658948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:09.658914    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9" podUID="791182e3-1f2f-400d-8577-fa86d6b74162" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:09.658948 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:09.658932    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg" podUID="08b09d3f-a61b-4812-b128-c4ea5cea422f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:09.862001 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:09.861971    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"]
Oct 30 21:56:09.862188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:09.862168    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy" containerID="cri-o://85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450" gracePeriod=30
Oct 30 21:56:10.029919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.029872    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs" podUID="c98af039-b120-4448-a7d9-a9e4a00ef226" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:10.029919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.029911    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb" podUID="49552f5c-97a2-4efb-b7e7-674879eb98cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:10.563481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.563439    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"]
Oct 30 21:56:10.563972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.563958    4918 cpu_manager.go:414] "containerMap: removing container" podUID="43d12812-6aa1-423f-acee-7ac62c91b0c3" containerName="revoke-keys"
Oct 30 21:56:10.564029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.563973    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d12812-6aa1-423f-acee-7ac62c91b0c3" containerName="revoke-keys"
Oct 30 21:56:10.564029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.563982    4918 cpu_manager.go:414] "containerMap: removing container" podUID="cc7098ea-8a7e-4286-89e2-c3f6bf14d92b" containerName="payload-pre-processing"
Oct 30 21:56:10.564029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.563989    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc7098ea-8a7e-4286-89e2-c3f6bf14d92b" containerName="payload-pre-processing"
Oct 30 21:56:10.564029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564003    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" containerName="istio-proxy"
Oct 30 21:56:10.564029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564008    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" containerName="istio-proxy"
Oct 30 21:56:10.564029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564018    4918 cpu_manager.go:414] "containerMap: removing container" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy"
Oct 30 21:56:10.564029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564023    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy"
Oct 30 21:56:10.564029 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564030    4918 cpu_manager.go:414] "containerMap: removing container" podUID="893a5d07-10fd-4b2f-a4a0-db937338f5db" containerName="payload-processing"
Oct 30 21:56:10.564276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564036    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893a5d07-10fd-4b2f-a4a0-db937338f5db" containerName="payload-processing"
Oct 30 21:56:10.564276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564051    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c84e9383-450d-49af-acf2-aa4ff741af9e" containerName="revoke-keys"
Oct 30 21:56:10.564276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564055    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84e9383-450d-49af-acf2-aa4ff741af9e" containerName="revoke-keys"
Oct 30 21:56:10.564276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564067    4918 cpu_manager.go:414] "containerMap: removing container" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy"
Oct 30 21:56:10.564276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564071    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy"
Oct 30 21:56:10.564276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564158    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" containerName="istio-proxy"
Oct 30 21:56:10.564276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564167    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy"
Oct 30 21:56:10.564276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564175    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="893a5d07-10fd-4b2f-a4a0-db937338f5db" containerName="payload-processing"
Oct 30 21:56:10.564276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564184    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84e9383-450d-49af-acf2-aa4ff741af9e" containerName="revoke-keys"
Oct 30 21:56:10.564276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564191    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7e7720-3b01-4e3b-bdfe-5c771d4346aa" containerName="istio-proxy"
Oct 30 21:56:10.564276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564199    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d12812-6aa1-423f-acee-7ac62c91b0c3" containerName="revoke-keys"
Oct 30 21:56:10.564276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.564204    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc7098ea-8a7e-4286-89e2-c3f6bf14d92b" containerName="payload-pre-processing"
Oct 30 21:56:10.569111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.569093    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:10.579292 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.579269    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:10.579450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.579340    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:56:10.579450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.579359    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:56:10.579587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.579474    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:56:10.582427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.582378    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"]
Oct 30 21:56:10.595135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.595105    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b\" (UID: \"42fb6e0e-26c4-4425-b604-fd43bf15f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:10.595370 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.595353    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b\" (UID: \"42fb6e0e-26c4-4425-b604-fd43bf15f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:10.595465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.595407    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mh96\" (UniqueName: \"kubernetes.io/projected/42fb6e0e-26c4-4425-b604-fd43bf15f226-kube-api-access-5mh96\") pod \"maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b\" (UID: \"42fb6e0e-26c4-4425-b604-fd43bf15f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:10.696270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.696235    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b\" (UID: \"42fb6e0e-26c4-4425-b604-fd43bf15f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:10.696270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.696273    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mh96\" (UniqueName: \"kubernetes.io/projected/42fb6e0e-26c4-4425-b604-fd43bf15f226-kube-api-access-5mh96\") pod \"maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b\" (UID: \"42fb6e0e-26c4-4425-b604-fd43bf15f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:10.696522 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.696323    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b\" (UID: \"42fb6e0e-26c4-4425-b604-fd43bf15f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:10.696995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.696965    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b\" (UID: \"42fb6e0e-26c4-4425-b604-fd43bf15f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:10.698852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.698833    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b\" (UID: \"42fb6e0e-26c4-4425-b604-fd43bf15f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:10.714030 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.714000    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mh96\" (UniqueName: \"kubernetes.io/projected/42fb6e0e-26c4-4425-b604-fd43bf15f226-kube-api-access-5mh96\") pod \"maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b\" (UID: \"42fb6e0e-26c4-4425-b604-fd43bf15f226\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:10.883457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:10.883428    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:11.111308 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.111275    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba" path="/var/lib/kubelet/pods/a9c62c0a-7427-4c23-bee1-9d3a7cd9b2ba/volumes"
Oct 30 21:56:11.245568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.245419    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"]
Oct 30 21:56:11.255444 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:56:11.248647    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fb6e0e_26c4_4425_b604_fd43bf15f226.slice/crio-bb1e71fc9d9fc30138386fee1dcb7485e9ce200988007b0b4c50c428b79480f5 WatchSource:0}: Error finding container bb1e71fc9d9fc30138386fee1dcb7485e9ce200988007b0b4c50c428b79480f5: Status 404 returned error can't find the container with id bb1e71fc9d9fc30138386fee1dcb7485e9ce200988007b0b4c50c428b79480f5
Oct 30 21:56:11.275288 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.275265    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"
Oct 30 21:56:11.312165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.312117    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b" event={"ID":"42fb6e0e-26c4-4425-b604-fd43bf15f226","Type":"ContainerStarted","Data":"bb1e71fc9d9fc30138386fee1dcb7485e9ce200988007b0b4c50c428b79480f5"}
Oct 30 21:56:11.313908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.313882    4918 generic.go:356] "Generic (PLEG): container finished" podID="d5cc4d1c-e96c-4df3-8b99-99257ffaf828" containerID="713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1" exitCode=0
Oct 30 21:56:11.314050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.313930    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px" event={"ID":"d5cc4d1c-e96c-4df3-8b99-99257ffaf828","Type":"ContainerDied","Data":"713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1"}
Oct 30 21:56:11.314050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.313938    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"
Oct 30 21:56:11.314050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.313975    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px" event={"ID":"d5cc4d1c-e96c-4df3-8b99-99257ffaf828","Type":"ContainerDied","Data":"cf416ee6e51d232fb801e7227180efb8ea892f7fc1a2d5ec7271a50a81a51863"}
Oct 30 21:56:11.314050 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.313997    4918 scope.go:122] "RemoveContainer" containerID="713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1"
Oct 30 21:56:11.327326 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.327305    4918 scope.go:122] "RemoveContainer" containerID="713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1"
Oct 30 21:56:11.327702 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:11.327676    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1\": container with ID starting with 713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1 not found: ID does not exist" containerID="713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1"
Oct 30 21:56:11.327804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.327712    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1"} err="failed to get container status \"713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1\": rpc error: code = NotFound desc = could not find container \"713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1\": container with ID starting with 713a2915a77b8696fd88195bc1d1d9d2e7e3a29ff747d5cf9eafbffd389a29a1 not found: ID does not exist"
Oct 30 21:56:11.407092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.407013    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-plugins-config-volume\") pod \"d5cc4d1c-e96c-4df3-8b99-99257ffaf828\" (UID: \"d5cc4d1c-e96c-4df3-8b99-99257ffaf828\") "
Oct 30 21:56:11.407258 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.407148    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-kube-api-access-tgq6t\" (UniqueName: \"kubernetes.io/projected/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-kube-api-access-tgq6t\") pod \"d5cc4d1c-e96c-4df3-8b99-99257ffaf828\" (UID: \"d5cc4d1c-e96c-4df3-8b99-99257ffaf828\") "
Oct 30 21:56:11.407481 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.407447    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-plugins-config-volume" pod "d5cc4d1c-e96c-4df3-8b99-99257ffaf828" (UID: "d5cc4d1c-e96c-4df3-8b99-99257ffaf828"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:11.436224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.436199    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"]
Oct 30 21:56:11.442815 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.442789    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bfc50e-kqhfd"]
Oct 30 21:56:11.506605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.506549    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-kube-api-access-tgq6t" pod "d5cc4d1c-e96c-4df3-8b99-99257ffaf828" (UID: "d5cc4d1c-e96c-4df3-8b99-99257ffaf828"). InnerVolumeSpecName "kube-api-access-tgq6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:11.508689 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.508660    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:11.508890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.508864    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgq6t\" (UniqueName: \"kubernetes.io/projected/d5cc4d1c-e96c-4df3-8b99-99257ffaf828-kube-api-access-tgq6t\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:11.641507 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.641476    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"]
Oct 30 21:56:11.653959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:11.653934    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e8d315-649bc8cf95-7d7px"]
Oct 30 21:56:12.324798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.324755    4918 generic.go:356] "Generic (PLEG): container finished" podID="42fb6e0e-26c4-4425-b604-fd43bf15f226" containerID="c3293e5cfa24a856a331eee63c5b834083b2bef9c33cf4812af7bff15c0105f2" exitCode=0
Oct 30 21:56:12.324988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.324845    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b" event={"ID":"42fb6e0e-26c4-4425-b604-fd43bf15f226","Type":"ContainerDied","Data":"c3293e5cfa24a856a331eee63c5b834083b2bef9c33cf4812af7bff15c0105f2"}
Oct 30 21:56:12.483659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.483628    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"]
Oct 30 21:56:12.484263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.484248    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d5cc4d1c-e96c-4df3-8b99-99257ffaf828" containerName="payload-pre-processing"
Oct 30 21:56:12.484312 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.484264    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cc4d1c-e96c-4df3-8b99-99257ffaf828" containerName="payload-pre-processing"
Oct 30 21:56:12.484375 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.484366    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5cc4d1c-e96c-4df3-8b99-99257ffaf828" containerName="payload-pre-processing"
Oct 30 21:56:12.488077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.488052    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:12.510611 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.510585    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"]
Oct 30 21:56:12.515823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.515804    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"]
Oct 30 21:56:12.516379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.515926    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:12.525440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.524796    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc834391-hh5m7\" (UID: \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:12.525440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.524912    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc834391-hh5m7\" (UID: \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:12.525440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.524945    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q29c7\" (UniqueName: \"kubernetes.io/projected/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-kube-api-access-q29c7\") pod \"maas-api-revoke-sub-fc834391-hh5m7\" (UID: \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:12.525440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.525047    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xvvz\" (UniqueName: \"kubernetes.io/projected/34140bfe-9c3d-4a45-a512-000a2702ae94-kube-api-access-5xvvz\") pod \"maas-api-revoke-sub-22d45c89-tmzfv\" (UID: \"34140bfe-9c3d-4a45-a512-000a2702ae94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:12.525440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.525123    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22d45c89-tmzfv\" (UID: \"34140bfe-9c3d-4a45-a512-000a2702ae94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:12.525440 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.525160    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22d45c89-tmzfv\" (UID: \"34140bfe-9c3d-4a45-a512-000a2702ae94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:12.546880 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.546849    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"]
Oct 30 21:56:12.626012 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.625979    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc834391-hh5m7\" (UID: \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:12.626177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.626069    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc834391-hh5m7\" (UID: \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:12.626177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.626092    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q29c7\" (UniqueName: \"kubernetes.io/projected/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-kube-api-access-q29c7\") pod \"maas-api-revoke-sub-fc834391-hh5m7\" (UID: \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:12.626177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.626173    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xvvz\" (UniqueName: \"kubernetes.io/projected/34140bfe-9c3d-4a45-a512-000a2702ae94-kube-api-access-5xvvz\") pod \"maas-api-revoke-sub-22d45c89-tmzfv\" (UID: \"34140bfe-9c3d-4a45-a512-000a2702ae94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:12.626458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.626422    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22d45c89-tmzfv\" (UID: \"34140bfe-9c3d-4a45-a512-000a2702ae94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:12.626620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.626482    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22d45c89-tmzfv\" (UID: \"34140bfe-9c3d-4a45-a512-000a2702ae94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:12.626896 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.626872    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc834391-hh5m7\" (UID: \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:12.627054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.627035    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22d45c89-tmzfv\" (UID: \"34140bfe-9c3d-4a45-a512-000a2702ae94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:12.628633 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.628608    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc834391-hh5m7\" (UID: \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:12.628848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.628828    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22d45c89-tmzfv\" (UID: \"34140bfe-9c3d-4a45-a512-000a2702ae94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:12.638911 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.638882    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q29c7\" (UniqueName: \"kubernetes.io/projected/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-kube-api-access-q29c7\") pod \"maas-api-revoke-sub-fc834391-hh5m7\" (UID: \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:12.639212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.639194    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xvvz\" (UniqueName: \"kubernetes.io/projected/34140bfe-9c3d-4a45-a512-000a2702ae94-kube-api-access-5xvvz\") pod \"maas-api-revoke-sub-22d45c89-tmzfv\" (UID: \"34140bfe-9c3d-4a45-a512-000a2702ae94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:12.817761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.817663    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:12.854271 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:12.853760    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:13.079667 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.079594    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459a5db7-a125-41cb-91c0-8493b9748fd0" path="/var/lib/kubelet/pods/459a5db7-a125-41cb-91c0-8493b9748fd0/volumes"
Oct 30 21:56:13.080049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.080036    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5cc4d1c-e96c-4df3-8b99-99257ffaf828" path="/var/lib/kubelet/pods/d5cc4d1c-e96c-4df3-8b99-99257ffaf828/volumes"
Oct 30 21:56:13.173137 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:56:13.173112    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34140bfe_9c3d_4a45_a512_000a2702ae94.slice/crio-3b2f7cd0e88a9c476e90f107fcfe3351bdad17ad3a8c6e84d717de11ecdcafb6 WatchSource:0}: Error finding container 3b2f7cd0e88a9c476e90f107fcfe3351bdad17ad3a8c6e84d717de11ecdcafb6: Status 404 returned error can't find the container with id 3b2f7cd0e88a9c476e90f107fcfe3351bdad17ad3a8c6e84d717de11ecdcafb6
Oct 30 21:56:13.173437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.173415    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"]
Oct 30 21:56:13.222541 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:56:13.222514    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6d36a7_18cf_4223_8c47_e199ecf31b3c.slice/crio-992a1dc32fc0b4e88717472cf2dc66c1ddc16ed645bee36df1c4a393091e37ee WatchSource:0}: Error finding container 992a1dc32fc0b4e88717472cf2dc66c1ddc16ed645bee36df1c4a393091e37ee: Status 404 returned error can't find the container with id 992a1dc32fc0b4e88717472cf2dc66c1ddc16ed645bee36df1c4a393091e37ee
Oct 30 21:56:13.222782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.222763    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"]
Oct 30 21:56:13.335899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.335816    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv" event={"ID":"34140bfe-9c3d-4a45-a512-000a2702ae94","Type":"ContainerStarted","Data":"3b2f7cd0e88a9c476e90f107fcfe3351bdad17ad3a8c6e84d717de11ecdcafb6"}
Oct 30 21:56:13.337274 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.337246    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7" event={"ID":"3b6d36a7-18cf-4223-8c47-e199ecf31b3c","Type":"ContainerStarted","Data":"992a1dc32fc0b4e88717472cf2dc66c1ddc16ed645bee36df1c4a393091e37ee"}
Oct 30 21:56:13.508853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.508826    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:13.537974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.537947    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-xks-ca\") pod \"42fb6e0e-26c4-4425-b604-fd43bf15f226\" (UID: \"42fb6e0e-26c4-4425-b604-fd43bf15f226\") "
Oct 30 21:56:13.538119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.538042    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-openshift-ca\") pod \"42fb6e0e-26c4-4425-b604-fd43bf15f226\" (UID: \"42fb6e0e-26c4-4425-b604-fd43bf15f226\") "
Oct 30 21:56:13.538119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.538078    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fb6e0e-26c4-4425-b604-fd43bf15f226-kube-api-access-5mh96\" (UniqueName: \"kubernetes.io/projected/42fb6e0e-26c4-4425-b604-fd43bf15f226-kube-api-access-5mh96\") pod \"42fb6e0e-26c4-4425-b604-fd43bf15f226\" (UID: \"42fb6e0e-26c4-4425-b604-fd43bf15f226\") "
Oct 30 21:56:13.538439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.538408    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-openshift-ca" pod "42fb6e0e-26c4-4425-b604-fd43bf15f226" (UID: "42fb6e0e-26c4-4425-b604-fd43bf15f226"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:13.538603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.538587    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:13.540192 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.540147    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-xks-ca" pod "42fb6e0e-26c4-4425-b604-fd43bf15f226" (UID: "42fb6e0e-26c4-4425-b604-fd43bf15f226"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:13.540320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.540241    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fb6e0e-26c4-4425-b604-fd43bf15f226-kube-api-access-5mh96" pod "42fb6e0e-26c4-4425-b604-fd43bf15f226" (UID: "42fb6e0e-26c4-4425-b604-fd43bf15f226"). InnerVolumeSpecName "kube-api-access-5mh96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:13.639764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.639738    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42fb6e0e-26c4-4425-b604-fd43bf15f226-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:13.639764 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.639765    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mh96\" (UniqueName: \"kubernetes.io/projected/42fb6e0e-26c4-4425-b604-fd43bf15f226-kube-api-access-5mh96\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:13.767529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:13.767499    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"]
Oct 30 21:56:14.269244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.269217    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"
Oct 30 21:56:14.349766 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.349732    4918 generic.go:356] "Generic (PLEG): container finished" podID="34140bfe-9c3d-4a45-a512-000a2702ae94" containerID="9f5adfb9ebcd1b4f79608319d9f3c9db2fe48e792b1dbdd5219b1c8a3cc4789e" exitCode=0
Oct 30 21:56:14.349966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.349842    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv" event={"ID":"34140bfe-9c3d-4a45-a512-000a2702ae94","Type":"ContainerDied","Data":"9f5adfb9ebcd1b4f79608319d9f3c9db2fe48e792b1dbdd5219b1c8a3cc4789e"}
Oct 30 21:56:14.352204 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.352174    4918 generic.go:356] "Generic (PLEG): container finished" podID="49552f5c-97a2-4efb-b7e7-674879eb98cf" containerID="eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8" exitCode=0
Oct 30 21:56:14.352420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.352223    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"
Oct 30 21:56:14.352420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.352261    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb" event={"ID":"49552f5c-97a2-4efb-b7e7-674879eb98cf","Type":"ContainerDied","Data":"eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8"}
Oct 30 21:56:14.352420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.352307    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb" event={"ID":"49552f5c-97a2-4efb-b7e7-674879eb98cf","Type":"ContainerDied","Data":"33363e1aa5f4c57e717857948d8456ec92aa4e3a4aca36ee43d20a24a74310f5"}
Oct 30 21:56:14.352420 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.352335    4918 scope.go:122] "RemoveContainer" containerID="eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8"
Oct 30 21:56:14.354998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.354968    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"
Oct 30 21:56:14.355116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.354976    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b" event={"ID":"42fb6e0e-26c4-4425-b604-fd43bf15f226","Type":"ContainerDied","Data":"bb1e71fc9d9fc30138386fee1dcb7485e9ce200988007b0b4c50c428b79480f5"}
Oct 30 21:56:14.355116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.355078    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb1e71fc9d9fc30138386fee1dcb7485e9ce200988007b0b4c50c428b79480f5"
Oct 30 21:56:14.356899 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.356872    4918 generic.go:356] "Generic (PLEG): container finished" podID="3b6d36a7-18cf-4223-8c47-e199ecf31b3c" containerID="0c053d3568d63601622c1a221e3442def6e530ef4994edcc799d3ddd5118dcc3" exitCode=0
Oct 30 21:56:14.357021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.356924    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7" event={"ID":"3b6d36a7-18cf-4223-8c47-e199ecf31b3c","Type":"ContainerDied","Data":"0c053d3568d63601622c1a221e3442def6e530ef4994edcc799d3ddd5118dcc3"}
Oct 30 21:56:14.358707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.358687    4918 generic.go:356] "Generic (PLEG): container finished" podID="c98af039-b120-4448-a7d9-a9e4a00ef226" containerID="166bc2dce5cb3cb357bd4566b40857caf66c6524f95beae268835119286a0be0" exitCode=0
Oct 30 21:56:14.358812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.358751    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs" event={"ID":"c98af039-b120-4448-a7d9-a9e4a00ef226","Type":"ContainerDied","Data":"166bc2dce5cb3cb357bd4566b40857caf66c6524f95beae268835119286a0be0"}
Oct 30 21:56:14.358812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.358789    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs" event={"ID":"c98af039-b120-4448-a7d9-a9e4a00ef226","Type":"ContainerDied","Data":"84d56f956fcc7ce3868de5b25b54b97fb9e86115389ee89f38e29c96f29a3eb6"}
Oct 30 21:56:14.358812 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.358812    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84d56f956fcc7ce3868de5b25b54b97fb9e86115389ee89f38e29c96f29a3eb6"
Oct 30 21:56:14.362977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.362954    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"
Oct 30 21:56:14.365995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.365973    4918 scope.go:122] "RemoveContainer" containerID="eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8"
Oct 30 21:56:14.366257 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:14.366239    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8\": container with ID starting with eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8 not found: ID does not exist" containerID="eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8"
Oct 30 21:56:14.366315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.366269    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8"} err="failed to get container status \"eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8\": rpc error: code = NotFound desc = could not find container \"eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8\": container with ID starting with eba3e4742ac60a79052ff5489ab79c0b7d8a67299f97ffc9c5d495e7acefbbd8 not found: ID does not exist"
Oct 30 21:56:14.386295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.386273    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49552f5c-97a2-4efb-b7e7-674879eb98cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49552f5c-97a2-4efb-b7e7-674879eb98cf-plugins-config-volume\") pod \"49552f5c-97a2-4efb-b7e7-674879eb98cf\" (UID: \"49552f5c-97a2-4efb-b7e7-674879eb98cf\") "
Oct 30 21:56:14.386458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.386445    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49552f5c-97a2-4efb-b7e7-674879eb98cf-kube-api-access-hsqh6\" (UniqueName: \"kubernetes.io/projected/49552f5c-97a2-4efb-b7e7-674879eb98cf-kube-api-access-hsqh6\") pod \"49552f5c-97a2-4efb-b7e7-674879eb98cf\" (UID: \"49552f5c-97a2-4efb-b7e7-674879eb98cf\") "
Oct 30 21:56:14.386659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.386641    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49552f5c-97a2-4efb-b7e7-674879eb98cf-plugins-config-volume" pod "49552f5c-97a2-4efb-b7e7-674879eb98cf" (UID: "49552f5c-97a2-4efb-b7e7-674879eb98cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:14.386845 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.386825    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49552f5c-97a2-4efb-b7e7-674879eb98cf-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:14.388726 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.388706    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49552f5c-97a2-4efb-b7e7-674879eb98cf-kube-api-access-hsqh6" pod "49552f5c-97a2-4efb-b7e7-674879eb98cf" (UID: "49552f5c-97a2-4efb-b7e7-674879eb98cf"). InnerVolumeSpecName "kube-api-access-hsqh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:14.488074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.488043    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98af039-b120-4448-a7d9-a9e4a00ef226-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98af039-b120-4448-a7d9-a9e4a00ef226-plugins-config-volume\") pod \"c98af039-b120-4448-a7d9-a9e4a00ef226\" (UID: \"c98af039-b120-4448-a7d9-a9e4a00ef226\") "
Oct 30 21:56:14.488241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.488125    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98af039-b120-4448-a7d9-a9e4a00ef226-kube-api-access-6wg47\" (UniqueName: \"kubernetes.io/projected/c98af039-b120-4448-a7d9-a9e4a00ef226-kube-api-access-6wg47\") pod \"c98af039-b120-4448-a7d9-a9e4a00ef226\" (UID: \"c98af039-b120-4448-a7d9-a9e4a00ef226\") "
Oct 30 21:56:14.488460 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.488428    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98af039-b120-4448-a7d9-a9e4a00ef226-plugins-config-volume" pod "c98af039-b120-4448-a7d9-a9e4a00ef226" (UID: "c98af039-b120-4448-a7d9-a9e4a00ef226"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:14.488642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.488623    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98af039-b120-4448-a7d9-a9e4a00ef226-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:14.488704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.488648    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsqh6\" (UniqueName: \"kubernetes.io/projected/49552f5c-97a2-4efb-b7e7-674879eb98cf-kube-api-access-hsqh6\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:14.490381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.490360    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98af039-b120-4448-a7d9-a9e4a00ef226-kube-api-access-6wg47" pod "c98af039-b120-4448-a7d9-a9e4a00ef226" (UID: "c98af039-b120-4448-a7d9-a9e4a00ef226"). InnerVolumeSpecName "kube-api-access-6wg47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:14.589793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.589759    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wg47\" (UniqueName: \"kubernetes.io/projected/c98af039-b120-4448-a7d9-a9e4a00ef226-kube-api-access-6wg47\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:14.709149 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.709119    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"]
Oct 30 21:56:14.730692 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:14.730659    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bfc50e-7cc49d7dbc-lzsnb"]
Oct 30 21:56:15.082408 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.082361    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49552f5c-97a2-4efb-b7e7-674879eb98cf" path="/var/lib/kubelet/pods/49552f5c-97a2-4efb-b7e7-674879eb98cf/volumes"
Oct 30 21:56:15.167337 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.167311    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:56:15.301738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.301704    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-certs\") pod \"1eb115eb-85bb-411f-b199-707dc9df13ba\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") "
Oct 30 21:56:15.302230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.301765    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-podinfo\") pod \"1eb115eb-85bb-411f-b199-707dc9df13ba\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") "
Oct 30 21:56:15.302230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.301799    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-token\" (UniqueName: \"kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-token\") pod \"1eb115eb-85bb-411f-b199-707dc9df13ba\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") "
Oct 30 21:56:15.302230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.301849    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-credential-socket\") pod \"1eb115eb-85bb-411f-b199-707dc9df13ba\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") "
Oct 30 21:56:15.302230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.301872    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-kube-api-access-d7jcr\" (UniqueName: \"kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-kube-api-access-d7jcr\") pod \"1eb115eb-85bb-411f-b199-707dc9df13ba\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") "
Oct 30 21:56:15.302230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.301968    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-envoy\") pod \"1eb115eb-85bb-411f-b199-707dc9df13ba\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") "
Oct 30 21:56:15.302230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302017    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-data\") pod \"1eb115eb-85bb-411f-b199-707dc9df13ba\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") "
Oct 30 21:56:15.302230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302096    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb115eb-85bb-411f-b199-707dc9df13ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb115eb-85bb-411f-b199-707dc9df13ba-istiod-ca-cert\") pod \"1eb115eb-85bb-411f-b199-707dc9df13ba\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") "
Oct 30 21:56:15.302230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302099    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-certs" pod "1eb115eb-85bb-411f-b199-707dc9df13ba" (UID: "1eb115eb-85bb-411f-b199-707dc9df13ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:15.302230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302147    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-socket\") pod \"1eb115eb-85bb-411f-b199-707dc9df13ba\" (UID: \"1eb115eb-85bb-411f-b199-707dc9df13ba\") "
Oct 30 21:56:15.302802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302258    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-credential-socket" pod "1eb115eb-85bb-411f-b199-707dc9df13ba" (UID: "1eb115eb-85bb-411f-b199-707dc9df13ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:15.302802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302328    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-data" pod "1eb115eb-85bb-411f-b199-707dc9df13ba" (UID: "1eb115eb-85bb-411f-b199-707dc9df13ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:15.302802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302528    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-socket" pod "1eb115eb-85bb-411f-b199-707dc9df13ba" (UID: "1eb115eb-85bb-411f-b199-707dc9df13ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:15.302802 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302538    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb115eb-85bb-411f-b199-707dc9df13ba-istiod-ca-cert" pod "1eb115eb-85bb-411f-b199-707dc9df13ba" (UID: "1eb115eb-85bb-411f-b199-707dc9df13ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:15.302964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302927    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eb115eb-85bb-411f-b199-707dc9df13ba-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.302964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302953    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.303040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302968    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.303040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302984    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.303040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.302997    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.304254 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.304226    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-podinfo" pod "1eb115eb-85bb-411f-b199-707dc9df13ba" (UID: "1eb115eb-85bb-411f-b199-707dc9df13ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:15.304539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.304520    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-kube-api-access-d7jcr" pod "1eb115eb-85bb-411f-b199-707dc9df13ba" (UID: "1eb115eb-85bb-411f-b199-707dc9df13ba"). InnerVolumeSpecName "kube-api-access-d7jcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:15.304776 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.304760    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-envoy" pod "1eb115eb-85bb-411f-b199-707dc9df13ba" (UID: "1eb115eb-85bb-411f-b199-707dc9df13ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:15.304824 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.304785    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-token" pod "1eb115eb-85bb-411f-b199-707dc9df13ba" (UID: "1eb115eb-85bb-411f-b199-707dc9df13ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:15.340263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.340210    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"]
Oct 30 21:56:15.367915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.367883    4918 generic.go:356] "Generic (PLEG): container finished" podID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerID="85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450" exitCode=0
Oct 30 21:56:15.368098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.367944    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"
Oct 30 21:56:15.368098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.367962    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" event={"ID":"1eb115eb-85bb-411f-b199-707dc9df13ba","Type":"ContainerDied","Data":"85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450"}
Oct 30 21:56:15.368098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.368049    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt" event={"ID":"1eb115eb-85bb-411f-b199-707dc9df13ba","Type":"ContainerDied","Data":"ee67b4d220fa9340ceb68a0875f68e1e1eb185013370ed7b3dc7e18bcc0b80d1"}
Oct 30 21:56:15.368098 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.368074    4918 scope.go:122] "RemoveContainer" containerID="85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450"
Oct 30 21:56:15.370106 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.370044    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"
Oct 30 21:56:15.382178 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.382153    4918 scope.go:122] "RemoveContainer" containerID="cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796"
Oct 30 21:56:15.395659 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.394537    4918 scope.go:122] "RemoveContainer" containerID="85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450"
Oct 30 21:56:15.395993 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:15.395963    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450\": container with ID starting with 85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450 not found: ID does not exist" containerID="85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450"
Oct 30 21:56:15.396075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.396008    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450"} err="failed to get container status \"85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450\": rpc error: code = NotFound desc = could not find container \"85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450\": container with ID starting with 85e3a6289b262da5261bcbabaa9f65bddf1dca64b2e9ae1fe24fafffeeee7450 not found: ID does not exist"
Oct 30 21:56:15.396075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.396036    4918 scope.go:122] "RemoveContainer" containerID="cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796"
Oct 30 21:56:15.396485 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:15.396458    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796\": container with ID starting with cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796 not found: ID does not exist" containerID="cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796"
Oct 30 21:56:15.396575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.396495    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796"} err="failed to get container status \"cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796\": rpc error: code = NotFound desc = could not find container \"cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796\": container with ID starting with cf598363bc9a2c0ef23cb127fbfc52fd0f8d12b1293d594344c86b704799c796 not found: ID does not exist"
Oct 30 21:56:15.406052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.404745    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.406052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.404767    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.406052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.404778    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7jcr\" (UniqueName: \"kubernetes.io/projected/1eb115eb-85bb-411f-b199-707dc9df13ba-kube-api-access-d7jcr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.406052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.404788    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eb115eb-85bb-411f-b199-707dc9df13ba-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.412869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.412676    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"]
Oct 30 21:56:15.422439 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.422323    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-bfc50e-5c5655d95f-z7rbs"]
Oct 30 21:56:15.449823 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.449787    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"]
Oct 30 21:56:15.454774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.454744    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-27804d-openshift-default-c89c65f7c-rxfxt"]
Oct 30 21:56:15.522500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.522474    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:15.535863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.535841    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:15.607022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.606989    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-xks-ca\") pod \"34140bfe-9c3d-4a45-a512-000a2702ae94\" (UID: \"34140bfe-9c3d-4a45-a512-000a2702ae94\") "
Oct 30 21:56:15.607213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.607083    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-openshift-ca\") pod \"34140bfe-9c3d-4a45-a512-000a2702ae94\" (UID: \"34140bfe-9c3d-4a45-a512-000a2702ae94\") "
Oct 30 21:56:15.607286 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.607233    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34140bfe-9c3d-4a45-a512-000a2702ae94-kube-api-access-5xvvz\" (UniqueName: \"kubernetes.io/projected/34140bfe-9c3d-4a45-a512-000a2702ae94-kube-api-access-5xvvz\") pod \"34140bfe-9c3d-4a45-a512-000a2702ae94\" (UID: \"34140bfe-9c3d-4a45-a512-000a2702ae94\") "
Oct 30 21:56:15.607405 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.607369    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-kube-api-access-q29c7\" (UniqueName: \"kubernetes.io/projected/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-kube-api-access-q29c7\") pod \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\" (UID: \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\") "
Oct 30 21:56:15.607492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.607427    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-openshift-ca\") pod \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\" (UID: \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\") "
Oct 30 21:56:15.607492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.607482    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-xks-ca\") pod \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\" (UID: \"3b6d36a7-18cf-4223-8c47-e199ecf31b3c\") "
Oct 30 21:56:15.607619 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.607505    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-openshift-ca" pod "34140bfe-9c3d-4a45-a512-000a2702ae94" (UID: "34140bfe-9c3d-4a45-a512-000a2702ae94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:15.607901 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.607879    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-openshift-ca" pod "3b6d36a7-18cf-4223-8c47-e199ecf31b3c" (UID: "3b6d36a7-18cf-4223-8c47-e199ecf31b3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:15.608017 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.608001    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.608091 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.608023    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.610341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.610313    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34140bfe-9c3d-4a45-a512-000a2702ae94-kube-api-access-5xvvz" pod "34140bfe-9c3d-4a45-a512-000a2702ae94" (UID: "34140bfe-9c3d-4a45-a512-000a2702ae94"). InnerVolumeSpecName "kube-api-access-5xvvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:15.610584 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.610559    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-xks-ca" pod "34140bfe-9c3d-4a45-a512-000a2702ae94" (UID: "34140bfe-9c3d-4a45-a512-000a2702ae94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:15.610700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.610584    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-xks-ca" pod "3b6d36a7-18cf-4223-8c47-e199ecf31b3c" (UID: "3b6d36a7-18cf-4223-8c47-e199ecf31b3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:15.610700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.610688    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-kube-api-access-q29c7" pod "3b6d36a7-18cf-4223-8c47-e199ecf31b3c" (UID: "3b6d36a7-18cf-4223-8c47-e199ecf31b3c"). InnerVolumeSpecName "kube-api-access-q29c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:15.709212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.709128    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xvvz\" (UniqueName: \"kubernetes.io/projected/34140bfe-9c3d-4a45-a512-000a2702ae94-kube-api-access-5xvvz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.709212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.709156    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q29c7\" (UniqueName: \"kubernetes.io/projected/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-kube-api-access-q29c7\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.709212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.709167    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6d36a7-18cf-4223-8c47-e199ecf31b3c-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:15.709212 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:15.709177    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34140bfe-9c3d-4a45-a512-000a2702ae94-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:16.380021 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:16.379988    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7"
Oct 30 21:56:16.380485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:16.380025    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc834391-hh5m7" event={"ID":"3b6d36a7-18cf-4223-8c47-e199ecf31b3c","Type":"ContainerDied","Data":"992a1dc32fc0b4e88717472cf2dc66c1ddc16ed645bee36df1c4a393091e37ee"}
Oct 30 21:56:16.380485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:16.380079    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="992a1dc32fc0b4e88717472cf2dc66c1ddc16ed645bee36df1c4a393091e37ee"
Oct 30 21:56:16.382677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:16.382659    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv"
Oct 30 21:56:16.382798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:16.382678    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22d45c89-tmzfv" event={"ID":"34140bfe-9c3d-4a45-a512-000a2702ae94","Type":"ContainerDied","Data":"3b2f7cd0e88a9c476e90f107fcfe3351bdad17ad3a8c6e84d717de11ecdcafb6"}
Oct 30 21:56:16.382798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:16.382723    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b2f7cd0e88a9c476e90f107fcfe3351bdad17ad3a8c6e84d717de11ecdcafb6"
Oct 30 21:56:17.082590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.082558    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" path="/var/lib/kubelet/pods/1eb115eb-85bb-411f-b199-707dc9df13ba/volumes"
Oct 30 21:56:17.083155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.083132    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98af039-b120-4448-a7d9-a9e4a00ef226" path="/var/lib/kubelet/pods/c98af039-b120-4448-a7d9-a9e4a00ef226/volumes"
Oct 30 21:56:17.177956 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.177934    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"
Oct 30 21:56:17.322951 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.322930    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"
Oct 30 21:56:17.326100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.326080    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b09d3f-a61b-4812-b128-c4ea5cea422f-kube-api-access-jldbm\" (UniqueName: \"kubernetes.io/projected/08b09d3f-a61b-4812-b128-c4ea5cea422f-kube-api-access-jldbm\") pod \"08b09d3f-a61b-4812-b128-c4ea5cea422f\" (UID: \"08b09d3f-a61b-4812-b128-c4ea5cea422f\") "
Oct 30 21:56:17.326183 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.326133    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b09d3f-a61b-4812-b128-c4ea5cea422f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b09d3f-a61b-4812-b128-c4ea5cea422f-plugins-config-volume\") pod \"08b09d3f-a61b-4812-b128-c4ea5cea422f\" (UID: \"08b09d3f-a61b-4812-b128-c4ea5cea422f\") "
Oct 30 21:56:17.326533 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.326511    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b09d3f-a61b-4812-b128-c4ea5cea422f-plugins-config-volume" pod "08b09d3f-a61b-4812-b128-c4ea5cea422f" (UID: "08b09d3f-a61b-4812-b128-c4ea5cea422f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:17.326649 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.326620    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b09d3f-a61b-4812-b128-c4ea5cea422f-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:17.328327 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.328309    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b09d3f-a61b-4812-b128-c4ea5cea422f-kube-api-access-jldbm" pod "08b09d3f-a61b-4812-b128-c4ea5cea422f" (UID: "08b09d3f-a61b-4812-b128-c4ea5cea422f"). InnerVolumeSpecName "kube-api-access-jldbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:17.391026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.390939    4918 generic.go:356] "Generic (PLEG): container finished" podID="08b09d3f-a61b-4812-b128-c4ea5cea422f" containerID="73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b" exitCode=0
Oct 30 21:56:17.391026 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.390984    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"
Oct 30 21:56:17.391589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.391027    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg" event={"ID":"08b09d3f-a61b-4812-b128-c4ea5cea422f","Type":"ContainerDied","Data":"73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b"}
Oct 30 21:56:17.391589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.391085    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg" event={"ID":"08b09d3f-a61b-4812-b128-c4ea5cea422f","Type":"ContainerDied","Data":"d230423931e29f31b47b063b3a371b2816724d5e84cd99e36d9e19da81ee0ae4"}
Oct 30 21:56:17.391589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.391114    4918 scope.go:122] "RemoveContainer" containerID="73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b"
Oct 30 21:56:17.392894 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.392832    4918 generic.go:356] "Generic (PLEG): container finished" podID="791182e3-1f2f-400d-8577-fa86d6b74162" containerID="a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11" exitCode=0
Oct 30 21:56:17.393037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.392862    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9" event={"ID":"791182e3-1f2f-400d-8577-fa86d6b74162","Type":"ContainerDied","Data":"a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11"}
Oct 30 21:56:17.393037 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.393011    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9" event={"ID":"791182e3-1f2f-400d-8577-fa86d6b74162","Type":"ContainerDied","Data":"2be111c99fbfe54876f5382a0f296b7eab46019710428b2b284bc364f765eb19"}
Oct 30 21:56:17.393176 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.392881    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"
Oct 30 21:56:17.404289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.404269    4918 scope.go:122] "RemoveContainer" containerID="73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b"
Oct 30 21:56:17.404627 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:17.404605    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b\": container with ID starting with 73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b not found: ID does not exist" containerID="73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b"
Oct 30 21:56:17.404743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.404641    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b"} err="failed to get container status \"73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b\": rpc error: code = NotFound desc = could not find container \"73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b\": container with ID starting with 73f29263ff945fc66d590e556cf4e21ec63c610594e7c827a2c3210458090f7b not found: ID does not exist"
Oct 30 21:56:17.404743 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.404665    4918 scope.go:122] "RemoveContainer" containerID="a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11"
Oct 30 21:56:17.415712 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.415694    4918 scope.go:122] "RemoveContainer" containerID="a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11"
Oct 30 21:56:17.416000 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:17.415982    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11\": container with ID starting with a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11 not found: ID does not exist" containerID="a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11"
Oct 30 21:56:17.416067 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.416007    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11"} err="failed to get container status \"a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11\": rpc error: code = NotFound desc = could not find container \"a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11\": container with ID starting with a83527ba63572494cec2198f322d4bfd4d4ce07ab6d9645f451874a0e17b4d11 not found: ID does not exist"
Oct 30 21:56:17.427151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.427132    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/791182e3-1f2f-400d-8577-fa86d6b74162-kube-api-access-vdm2j\" (UniqueName: \"kubernetes.io/projected/791182e3-1f2f-400d-8577-fa86d6b74162-kube-api-access-vdm2j\") pod \"791182e3-1f2f-400d-8577-fa86d6b74162\" (UID: \"791182e3-1f2f-400d-8577-fa86d6b74162\") "
Oct 30 21:56:17.427305 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.427289    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/791182e3-1f2f-400d-8577-fa86d6b74162-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791182e3-1f2f-400d-8577-fa86d6b74162-plugins-config-volume\") pod \"791182e3-1f2f-400d-8577-fa86d6b74162\" (UID: \"791182e3-1f2f-400d-8577-fa86d6b74162\") "
Oct 30 21:56:17.427674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.427655    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791182e3-1f2f-400d-8577-fa86d6b74162-plugins-config-volume" pod "791182e3-1f2f-400d-8577-fa86d6b74162" (UID: "791182e3-1f2f-400d-8577-fa86d6b74162"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:17.427790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.427738    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/791182e3-1f2f-400d-8577-fa86d6b74162-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:17.427790 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.427753    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jldbm\" (UniqueName: \"kubernetes.io/projected/08b09d3f-a61b-4812-b128-c4ea5cea422f-kube-api-access-jldbm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:17.429253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.429232    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791182e3-1f2f-400d-8577-fa86d6b74162-kube-api-access-vdm2j" pod "791182e3-1f2f-400d-8577-fa86d6b74162" (UID: "791182e3-1f2f-400d-8577-fa86d6b74162"). InnerVolumeSpecName "kube-api-access-vdm2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:17.434142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.434124    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"]
Oct 30 21:56:17.445548 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.445527    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-27804d-d45bd7c55-c6dtg"]
Oct 30 21:56:17.528915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.528887    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdm2j\" (UniqueName: \"kubernetes.io/projected/791182e3-1f2f-400d-8577-fa86d6b74162-kube-api-access-vdm2j\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:17.722143 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.722112    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"]
Oct 30 21:56:17.735148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:17.735114    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-27804d-5799785d6c-z99m9"]
Oct 30 21:56:18.768674 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:18.768582    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6" podUID="9938d41e-a3ab-439c-a2ed-38673adc54c9" containerName="payload-processing" containerID="cri-o://7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1" gracePeriod=25
Oct 30 21:56:19.024621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.024554    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"
Oct 30 21:56:19.078492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.078461    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b09d3f-a61b-4812-b128-c4ea5cea422f" path="/var/lib/kubelet/pods/08b09d3f-a61b-4812-b128-c4ea5cea422f/volumes"
Oct 30 21:56:19.078907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.078892    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="791182e3-1f2f-400d-8577-fa86d6b74162" path="/var/lib/kubelet/pods/791182e3-1f2f-400d-8577-fa86d6b74162/volumes"
Oct 30 21:56:19.165925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.165878    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9938d41e-a3ab-439c-a2ed-38673adc54c9-kube-api-access-pp8zq\" (UniqueName: \"kubernetes.io/projected/9938d41e-a3ab-439c-a2ed-38673adc54c9-kube-api-access-pp8zq\") pod \"9938d41e-a3ab-439c-a2ed-38673adc54c9\" (UID: \"9938d41e-a3ab-439c-a2ed-38673adc54c9\") "
Oct 30 21:56:19.166133 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.166118    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9938d41e-a3ab-439c-a2ed-38673adc54c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9938d41e-a3ab-439c-a2ed-38673adc54c9-plugins-config-volume\") pod \"9938d41e-a3ab-439c-a2ed-38673adc54c9\" (UID: \"9938d41e-a3ab-439c-a2ed-38673adc54c9\") "
Oct 30 21:56:19.166775 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.166741    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9938d41e-a3ab-439c-a2ed-38673adc54c9-plugins-config-volume" pod "9938d41e-a3ab-439c-a2ed-38673adc54c9" (UID: "9938d41e-a3ab-439c-a2ed-38673adc54c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:19.178601 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.178563    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9938d41e-a3ab-439c-a2ed-38673adc54c9-kube-api-access-pp8zq" pod "9938d41e-a3ab-439c-a2ed-38673adc54c9" (UID: "9938d41e-a3ab-439c-a2ed-38673adc54c9"). InnerVolumeSpecName "kube-api-access-pp8zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:19.268022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.267992    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9938d41e-a3ab-439c-a2ed-38673adc54c9-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:19.268022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.268019    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8zq\" (UniqueName: \"kubernetes.io/projected/9938d41e-a3ab-439c-a2ed-38673adc54c9-kube-api-access-pp8zq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:19.414871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.414777    4918 generic.go:356] "Generic (PLEG): container finished" podID="9938d41e-a3ab-439c-a2ed-38673adc54c9" containerID="7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1" exitCode=0
Oct 30 21:56:19.414871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.414825    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"
Oct 30 21:56:19.415086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.414864    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6" event={"ID":"9938d41e-a3ab-439c-a2ed-38673adc54c9","Type":"ContainerDied","Data":"7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1"}
Oct 30 21:56:19.415086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.414923    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6" event={"ID":"9938d41e-a3ab-439c-a2ed-38673adc54c9","Type":"ContainerDied","Data":"5e78864ef77a7b23b9f28988cf10f875696e2a1e140732d467fda0bb9aab2ed1"}
Oct 30 21:56:19.415086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.414951    4918 scope.go:122] "RemoveContainer" containerID="7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1"
Oct 30 21:56:19.426076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.426059    4918 scope.go:122] "RemoveContainer" containerID="7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1"
Oct 30 21:56:19.426371 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:19.426347    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1\": container with ID starting with 7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1 not found: ID does not exist" containerID="7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1"
Oct 30 21:56:19.426503 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.426383    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1"} err="failed to get container status \"7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1\": rpc error: code = NotFound desc = could not find container \"7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1\": container with ID starting with 7da4a50290aac0d6e6af1c164c956c9cf74f7802296f9f624a7d23d72440b8d1 not found: ID does not exist"
Oct 30 21:56:19.460643 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.460609    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"]
Oct 30 21:56:19.467723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:19.467695    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fb2b58-55858586b9-r5cg6"]
Oct 30 21:56:20.392357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:20.392322    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"]
Oct 30 21:56:20.435997 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:20.435968    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fb2b58-xrggn"]
Oct 30 21:56:20.822506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:20.822473    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"]
Oct 30 21:56:20.867383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:20.867344    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"]
Oct 30 21:56:21.079306 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:21.079229    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d12812-6aa1-423f-acee-7ac62c91b0c3" path="/var/lib/kubelet/pods/43d12812-6aa1-423f-acee-7ac62c91b0c3/volumes"
Oct 30 21:56:21.079765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:21.079745    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9938d41e-a3ab-439c-a2ed-38673adc54c9" path="/var/lib/kubelet/pods/9938d41e-a3ab-439c-a2ed-38673adc54c9/volumes"
Oct 30 21:56:22.087843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.087780    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main" containerID="cri-o://c4f354300479e1180fac0f1195f25b0c107b127e9265702a990ba92dba149be0" gracePeriod=45
Oct 30 21:56:22.104959 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.104923    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main" containerID="cri-o://c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64" gracePeriod=45
Oct 30 21:56:22.115753 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.115724    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main" containerID="cri-o://d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587" gracePeriod=45
Oct 30 21:56:22.408723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.408697    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:56:22.412107 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.412080    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:56:22.441474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.441440    4918 generic.go:356] "Generic (PLEG): container finished" podID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerID="c4f354300479e1180fac0f1195f25b0c107b127e9265702a990ba92dba149be0" exitCode=0
Oct 30 21:56:22.441621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.441493    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" event={"ID":"4a2f5787-d041-4361-aba9-f8c765ea3909","Type":"ContainerDied","Data":"c4f354300479e1180fac0f1195f25b0c107b127e9265702a990ba92dba149be0"}
Oct 30 21:56:22.441621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.441547    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6" event={"ID":"4a2f5787-d041-4361-aba9-f8c765ea3909","Type":"ContainerDied","Data":"0f3d7d9423f4e2619ab0613967728b5639b2abe1104f7305f7674dbac413f2ba"}
Oct 30 21:56:22.441621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.441565    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f3d7d9423f4e2619ab0613967728b5639b2abe1104f7305f7674dbac413f2ba"
Oct 30 21:56:22.442708 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.442687    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:56:22.442889 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.442873    4918 generic.go:356] "Generic (PLEG): container finished" podID="1417521f-e732-4114-b645-b51b1c731cd6" containerID="d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587" exitCode=0
Oct 30 21:56:22.442941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.442909    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"
Oct 30 21:56:22.442989 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.442947    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" event={"ID":"1417521f-e732-4114-b645-b51b1c731cd6","Type":"ContainerDied","Data":"d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587"}
Oct 30 21:56:22.442989 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.442978    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj" event={"ID":"1417521f-e732-4114-b645-b51b1c731cd6","Type":"ContainerDied","Data":"51c492a4d36dacb218f95405b1113bc0c8b90404f481e19cba353bb8c023953b"}
Oct 30 21:56:22.443104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.442994    4918 scope.go:122] "RemoveContainer" containerID="d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587"
Oct 30 21:56:22.444280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.444257    4918 generic.go:356] "Generic (PLEG): container finished" podID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerID="c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64" exitCode=0
Oct 30 21:56:22.444377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.444295    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"
Oct 30 21:56:22.444377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.444336    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" event={"ID":"08ff176b-aa57-4514-a607-0b1513ca3bb0","Type":"ContainerDied","Data":"c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64"}
Oct 30 21:56:22.444377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.444370    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn" event={"ID":"08ff176b-aa57-4514-a607-0b1513ca3bb0","Type":"ContainerDied","Data":"5340b661f2573963b090b7ef7ffa6940fcf4a1132728ba72e7a09be460de002d"}
Oct 30 21:56:22.456547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.456526    4918 scope.go:122] "RemoveContainer" containerID="d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587"
Oct 30 21:56:22.457011 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:22.456971    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587\": container with ID starting with d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587 not found: ID does not exist" containerID="d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587"
Oct 30 21:56:22.457117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.457004    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587"} err="failed to get container status \"d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587\": rpc error: code = NotFound desc = could not find container \"d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587\": container with ID starting with d27379e6856891c2587e3cbfa7b494148aec96ea1c470488be1d32bc13904587 not found: ID does not exist"
Oct 30 21:56:22.457117 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.457027    4918 scope.go:122] "RemoveContainer" containerID="c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64"
Oct 30 21:56:22.473527 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.473502    4918 scope.go:122] "RemoveContainer" containerID="c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64"
Oct 30 21:56:22.473860 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:22.473838    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64\": container with ID starting with c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64 not found: ID does not exist" containerID="c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64"
Oct 30 21:56:22.473957 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.473868    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64"} err="failed to get container status \"c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64\": rpc error: code = NotFound desc = could not find container \"c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64\": container with ID starting with c9e100c3a0f2068b56e9f865253f50ca3f852c158113232f809eeb927a21af64 not found: ID does not exist"
Oct 30 21:56:22.512062 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512023    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-tmp-dir\") pod \"4a2f5787-d041-4361-aba9-f8c765ea3909\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") "
Oct 30 21:56:22.512247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512116    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-tmp-dir\") pod \"08ff176b-aa57-4514-a607-0b1513ca3bb0\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") "
Oct 30 21:56:22.512247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512161    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-home\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-home\") pod \"08ff176b-aa57-4514-a607-0b1513ca3bb0\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") "
Oct 30 21:56:22.512247 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512197    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-model-cache\") pod \"4a2f5787-d041-4361-aba9-f8c765ea3909\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") "
Oct 30 21:56:22.512457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512244    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-model-cache\") pod \"1417521f-e732-4114-b645-b51b1c731cd6\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") "
Oct 30 21:56:22.512457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512279    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-dshm\") pod \"1417521f-e732-4114-b645-b51b1c731cd6\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") "
Oct 30 21:56:22.512457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512385    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-tmp-dir" pod "4a2f5787-d041-4361-aba9-f8c765ea3909" (UID: "4a2f5787-d041-4361-aba9-f8c765ea3909"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:22.512457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512380    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-tmp-dir\") pod \"1417521f-e732-4114-b645-b51b1c731cd6\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") "
Oct 30 21:56:22.512680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512485    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1417521f-e732-4114-b645-b51b1c731cd6-kube-api-access-zrxl9\" (UniqueName: \"kubernetes.io/projected/1417521f-e732-4114-b645-b51b1c731cd6-kube-api-access-zrxl9\") pod \"1417521f-e732-4114-b645-b51b1c731cd6\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") "
Oct 30 21:56:22.512680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512533    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-home\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-home\") pod \"1417521f-e732-4114-b645-b51b1c731cd6\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") "
Oct 30 21:56:22.512680 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512647    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ff176b-aa57-4514-a607-0b1513ca3bb0-tls-certs\" (UniqueName: \"kubernetes.io/secret/08ff176b-aa57-4514-a607-0b1513ca3bb0-tls-certs\") pod \"08ff176b-aa57-4514-a607-0b1513ca3bb0\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") "
Oct 30 21:56:22.512897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512715    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-home\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-home\") pod \"4a2f5787-d041-4361-aba9-f8c765ea3909\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") "
Oct 30 21:56:22.512897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512778    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2f5787-d041-4361-aba9-f8c765ea3909-kube-api-access-6lz2l\" (UniqueName: \"kubernetes.io/projected/4a2f5787-d041-4361-aba9-f8c765ea3909-kube-api-access-6lz2l\") pod \"4a2f5787-d041-4361-aba9-f8c765ea3909\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") "
Oct 30 21:56:22.512897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512815    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1417521f-e732-4114-b645-b51b1c731cd6-tls-certs\" (UniqueName: \"kubernetes.io/secret/1417521f-e732-4114-b645-b51b1c731cd6-tls-certs\") pod \"1417521f-e732-4114-b645-b51b1c731cd6\" (UID: \"1417521f-e732-4114-b645-b51b1c731cd6\") "
Oct 30 21:56:22.512897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512858    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-dshm\") pod \"4a2f5787-d041-4361-aba9-f8c765ea3909\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") "
Oct 30 21:56:22.513151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512896    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a2f5787-d041-4361-aba9-f8c765ea3909-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2f5787-d041-4361-aba9-f8c765ea3909-tls-certs\") pod \"4a2f5787-d041-4361-aba9-f8c765ea3909\" (UID: \"4a2f5787-d041-4361-aba9-f8c765ea3909\") "
Oct 30 21:56:22.513151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512961    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-dshm\") pod \"08ff176b-aa57-4514-a607-0b1513ca3bb0\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") "
Oct 30 21:56:22.513151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512995    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-model-cache\") pod \"08ff176b-aa57-4514-a607-0b1513ca3bb0\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") "
Oct 30 21:56:22.513151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.513060    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ff176b-aa57-4514-a607-0b1513ca3bb0-kube-api-access-z54gg\" (UniqueName: \"kubernetes.io/projected/08ff176b-aa57-4514-a607-0b1513ca3bb0-kube-api-access-z54gg\") pod \"08ff176b-aa57-4514-a607-0b1513ca3bb0\" (UID: \"08ff176b-aa57-4514-a607-0b1513ca3bb0\") "
Oct 30 21:56:22.513884 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.513858    4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.514059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512651    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-tmp-dir" pod "1417521f-e732-4114-b645-b51b1c731cd6" (UID: "1417521f-e732-4114-b645-b51b1c731cd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:22.514059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512829    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-tmp-dir" pod "08ff176b-aa57-4514-a607-0b1513ca3bb0" (UID: "08ff176b-aa57-4514-a607-0b1513ca3bb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:22.514059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.512991    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-home" pod "08ff176b-aa57-4514-a607-0b1513ca3bb0" (UID: "08ff176b-aa57-4514-a607-0b1513ca3bb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:22.514059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.513153    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-model-cache" pod "4a2f5787-d041-4361-aba9-f8c765ea3909" (UID: "4a2f5787-d041-4361-aba9-f8c765ea3909"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:22.514059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.513311    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-model-cache" pod "1417521f-e732-4114-b645-b51b1c731cd6" (UID: "1417521f-e732-4114-b645-b51b1c731cd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:22.514914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.514686    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-home" pod "4a2f5787-d041-4361-aba9-f8c765ea3909" (UID: "4a2f5787-d041-4361-aba9-f8c765ea3909"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:22.519724 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.519681    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ff176b-aa57-4514-a607-0b1513ca3bb0-kube-api-access-z54gg" pod "08ff176b-aa57-4514-a607-0b1513ca3bb0" (UID: "08ff176b-aa57-4514-a607-0b1513ca3bb0"). InnerVolumeSpecName "kube-api-access-z54gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:22.519963 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.519937    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-model-cache" pod "08ff176b-aa57-4514-a607-0b1513ca3bb0" (UID: "08ff176b-aa57-4514-a607-0b1513ca3bb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:22.520188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.520163    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-home" pod "1417521f-e732-4114-b645-b51b1c731cd6" (UID: "1417521f-e732-4114-b645-b51b1c731cd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:22.521374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.520371    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-dshm" pod "08ff176b-aa57-4514-a607-0b1513ca3bb0" (UID: "08ff176b-aa57-4514-a607-0b1513ca3bb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:22.522761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.522718    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-dshm" pod "1417521f-e732-4114-b645-b51b1c731cd6" (UID: "1417521f-e732-4114-b645-b51b1c731cd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:22.522761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.522717    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ff176b-aa57-4514-a607-0b1513ca3bb0-tls-certs" pod "08ff176b-aa57-4514-a607-0b1513ca3bb0" (UID: "08ff176b-aa57-4514-a607-0b1513ca3bb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:22.522761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.522735    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2f5787-d041-4361-aba9-f8c765ea3909-tls-certs" pod "4a2f5787-d041-4361-aba9-f8c765ea3909" (UID: "4a2f5787-d041-4361-aba9-f8c765ea3909"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:22.523620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.522923    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-dshm" pod "4a2f5787-d041-4361-aba9-f8c765ea3909" (UID: "4a2f5787-d041-4361-aba9-f8c765ea3909"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:22.523620 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.523155    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2f5787-d041-4361-aba9-f8c765ea3909-kube-api-access-6lz2l" pod "4a2f5787-d041-4361-aba9-f8c765ea3909" (UID: "4a2f5787-d041-4361-aba9-f8c765ea3909"). InnerVolumeSpecName "kube-api-access-6lz2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:22.525623 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.525590    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1417521f-e732-4114-b645-b51b1c731cd6-kube-api-access-zrxl9" pod "1417521f-e732-4114-b645-b51b1c731cd6" (UID: "1417521f-e732-4114-b645-b51b1c731cd6"). InnerVolumeSpecName "kube-api-access-zrxl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:22.526524 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.526497    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1417521f-e732-4114-b645-b51b1c731cd6-tls-certs" pod "1417521f-e732-4114-b645-b51b1c731cd6" (UID: "1417521f-e732-4114-b645-b51b1c731cd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616618    4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2f5787-d041-4361-aba9-f8c765ea3909-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616652    4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616674    4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616690    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z54gg\" (UniqueName: \"kubernetes.io/projected/08ff176b-aa57-4514-a607-0b1513ca3bb0-kube-api-access-z54gg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616705    4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616720    4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ff176b-aa57-4514-a607-0b1513ca3bb0-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616738    4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616754    4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616767    4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616781    4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616798    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxl9\" (UniqueName: \"kubernetes.io/projected/1417521f-e732-4114-b645-b51b1c731cd6-kube-api-access-zrxl9\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616811    4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1417521f-e732-4114-b645-b51b1c731cd6-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616826    4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ff176b-aa57-4514-a607-0b1513ca3bb0-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616838    4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616857    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lz2l\" (UniqueName: \"kubernetes.io/projected/4a2f5787-d041-4361-aba9-f8c765ea3909-kube-api-access-6lz2l\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616871    4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1417521f-e732-4114-b645-b51b1c731cd6-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.617931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.616884    4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2f5787-d041-4361-aba9-f8c765ea3909-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:22.781985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.781955    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"]
Oct 30 21:56:22.799325 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.799295    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/premium-w4-3011c6-kserve-d4985fdbb-cb2tj"]
Oct 30 21:56:22.815627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.815602    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"]
Oct 30 21:56:22.828425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:22.828379    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/distinct-w4-3011c6-kserve-65785b867c-vc6pn"]
Oct 30 21:56:23.077525 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:23.077490    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" path="/var/lib/kubelet/pods/08ff176b-aa57-4514-a607-0b1513ca3bb0/volumes"
Oct 30 21:56:23.077946 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:23.077927    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1417521f-e732-4114-b645-b51b1c731cd6" path="/var/lib/kubelet/pods/1417521f-e732-4114-b645-b51b1c731cd6/volumes"
Oct 30 21:56:23.232329 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:23.232289    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"]
Oct 30 21:56:23.232832 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:23.232544    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" podUID="197109cc-f204-4d13-bf73-02172b3003ec" containerName="istio-proxy" containerID="cri-o://92454911f832d44911996bc3c7e62bc45b068b14d52175e06e70611130a9fff5" gracePeriod=30
Oct 30 21:56:23.454587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:23.454509    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"
Oct 30 21:56:23.489944 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:23.489916    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"]
Oct 30 21:56:23.511848 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:23.511823    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3011c6/distinct-2-w4-3011c6-kserve-696cfcc4f4-2j6c6"]
Oct 30 21:56:25.077787 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:25.077757    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" path="/var/lib/kubelet/pods/4a2f5787-d041-4361-aba9-f8c765ea3909/volumes"
Oct 30 21:56:25.823676 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:25.823633    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm" podUID="ff7f4bf0-e541-4893-a000-027b14e42b74" containerName="payload-processing" containerID="cri-o://bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c" gracePeriod=25
Oct 30 21:56:25.868215 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:25.868174    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7" podUID="a13a507e-48ca-4b4a-90dd-72f30316c75d" containerName="payload-pre-processing" containerID="cri-o://92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342" gracePeriod=25
Oct 30 21:56:26.076734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:26.076375    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7" podUID="a13a507e-48ca-4b4a-90dd-72f30316c75d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:26.077015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:26.076980    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm" podUID="ff7f4bf0-e541-4893-a000-027b14e42b74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:26.253547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:26.253516    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"]
Oct 30 21:56:26.266033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:26.265997    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3011c6-gccd9"]
Oct 30 21:56:27.078320 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:27.078284    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84e9383-450d-49af-acf2-aa4ff741af9e" path="/var/lib/kubelet/pods/c84e9383-450d-49af-acf2-aa4ff741af9e/volumes"
Oct 30 21:56:28.398487 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398453    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"]
Oct 30 21:56:28.398914 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398905    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c98af039-b120-4448-a7d9-a9e4a00ef226" containerName="payload-pre-processing"
Oct 30 21:56:28.398985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398915    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98af039-b120-4448-a7d9-a9e4a00ef226" containerName="payload-pre-processing"
Oct 30 21:56:28.398985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398935    4918 cpu_manager.go:414] "containerMap: removing container" podUID="08b09d3f-a61b-4812-b128-c4ea5cea422f" containerName="payload-processing"
Oct 30 21:56:28.398985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398941    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b09d3f-a61b-4812-b128-c4ea5cea422f" containerName="payload-processing"
Oct 30 21:56:28.398985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398954    4918 cpu_manager.go:414] "containerMap: removing container" podUID="49552f5c-97a2-4efb-b7e7-674879eb98cf" containerName="payload-processing"
Oct 30 21:56:28.398985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398959    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49552f5c-97a2-4efb-b7e7-674879eb98cf" containerName="payload-processing"
Oct 30 21:56:28.398985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398968    4918 cpu_manager.go:414] "containerMap: removing container" podUID="42fb6e0e-26c4-4425-b604-fd43bf15f226" containerName="revoke-keys"
Oct 30 21:56:28.398985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398973    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fb6e0e-26c4-4425-b604-fd43bf15f226" containerName="revoke-keys"
Oct 30 21:56:28.398985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398979    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy"
Oct 30 21:56:28.398985 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398983    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398991    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.398995    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399007    4918 cpu_manager.go:414] "containerMap: removing container" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399011    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399017    4918 cpu_manager.go:414] "containerMap: removing container" podUID="34140bfe-9c3d-4a45-a512-000a2702ae94" containerName="revoke-keys"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399021    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34140bfe-9c3d-4a45-a512-000a2702ae94" containerName="revoke-keys"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399028    4918 cpu_manager.go:414] "containerMap: removing container" podUID="791182e3-1f2f-400d-8577-fa86d6b74162" containerName="payload-pre-processing"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399033    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="791182e3-1f2f-400d-8577-fa86d6b74162" containerName="payload-pre-processing"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399040    4918 cpu_manager.go:414] "containerMap: removing container" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399044    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399060    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9938d41e-a3ab-439c-a2ed-38673adc54c9" containerName="payload-processing"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399065    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9938d41e-a3ab-439c-a2ed-38673adc54c9" containerName="payload-processing"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399072    4918 cpu_manager.go:414] "containerMap: removing container" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399076    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399083    4918 cpu_manager.go:414] "containerMap: removing container" podUID="3b6d36a7-18cf-4223-8c47-e199ecf31b3c" containerName="revoke-keys"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399087    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6d36a7-18cf-4223-8c47-e199ecf31b3c" containerName="revoke-keys"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399151    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b09d3f-a61b-4812-b128-c4ea5cea422f" containerName="payload-processing"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399157    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="49552f5c-97a2-4efb-b7e7-674879eb98cf" containerName="payload-processing"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399163    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fb6e0e-26c4-4425-b604-fd43bf15f226" containerName="revoke-keys"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399170    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399175    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb115eb-85bb-411f-b199-707dc9df13ba" containerName="istio-proxy"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399181    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2f5787-d041-4361-aba9-f8c765ea3909" containerName="main"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399187    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="34140bfe-9c3d-4a45-a512-000a2702ae94" containerName="revoke-keys"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399193    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="791182e3-1f2f-400d-8577-fa86d6b74162" containerName="payload-pre-processing"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399198    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ff176b-aa57-4514-a607-0b1513ca3bb0" containerName="main"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399205    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9938d41e-a3ab-439c-a2ed-38673adc54c9" containerName="payload-processing"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399211    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="1417521f-e732-4114-b645-b51b1c731cd6" containerName="main"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399217    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6d36a7-18cf-4223-8c47-e199ecf31b3c" containerName="revoke-keys"
Oct 30 21:56:28.399343 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.399222    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98af039-b120-4448-a7d9-a9e4a00ef226" containerName="payload-pre-processing"
Oct 30 21:56:28.402429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.402413    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:28.410078 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.410052    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"]
Oct 30 21:56:28.412998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.412964    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:56:28.414163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.414089    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:56:28.414163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.414122    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:56:28.414163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.414133    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:28.428430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.425572    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"]
Oct 30 21:56:28.428430 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.425682    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:28.445248 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.445218    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"]
Oct 30 21:56:28.486031 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.485997    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-379bb5b6-l6d4p\" (UID: \"b209119c-222d-444f-a597-43fdc39d42ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:28.486244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.486053    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-379bb5b6-l6d4p\" (UID: \"b209119c-222d-444f-a597-43fdc39d42ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:28.486244 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.486200    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93653586-m58wl\" (UID: \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:28.486450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.486422    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqcfj\" (UniqueName: \"kubernetes.io/projected/b209119c-222d-444f-a597-43fdc39d42ac-kube-api-access-bqcfj\") pod \"maas-api-revoke-sub-379bb5b6-l6d4p\" (UID: \"b209119c-222d-444f-a597-43fdc39d42ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:28.486559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.486478    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thz8q\" (UniqueName: \"kubernetes.io/projected/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-kube-api-access-thz8q\") pod \"maas-api-revoke-sub-93653586-m58wl\" (UID: \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:28.486630 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.486563    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93653586-m58wl\" (UID: \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:28.496679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.496651    4918 generic.go:356] "Generic (PLEG): container finished" podID="197109cc-f204-4d13-bf73-02172b3003ec" containerID="92454911f832d44911996bc3c7e62bc45b068b14d52175e06e70611130a9fff5" exitCode=0
Oct 30 21:56:28.496841 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.496731    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" event={"ID":"197109cc-f204-4d13-bf73-02172b3003ec","Type":"ContainerDied","Data":"92454911f832d44911996bc3c7e62bc45b068b14d52175e06e70611130a9fff5"}
Oct 30 21:56:28.522163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.522143    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:56:28.588135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588104    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-istio-token\" (UniqueName: \"kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-istio-token\") pod \"197109cc-f204-4d13-bf73-02172b3003ec\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") "
Oct 30 21:56:28.588135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588141    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/197109cc-f204-4d13-bf73-02172b3003ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197109cc-f204-4d13-bf73-02172b3003ec-istio-podinfo\") pod \"197109cc-f204-4d13-bf73-02172b3003ec\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") "
Oct 30 21:56:28.588414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588219    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-credential-socket\") pod \"197109cc-f204-4d13-bf73-02172b3003ec\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") "
Oct 30 21:56:28.588414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588242    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-socket\") pod \"197109cc-f204-4d13-bf73-02172b3003ec\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") "
Oct 30 21:56:28.588414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588271    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/197109cc-f204-4d13-bf73-02172b3003ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197109cc-f204-4d13-bf73-02172b3003ec-istiod-ca-cert\") pod \"197109cc-f204-4d13-bf73-02172b3003ec\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") "
Oct 30 21:56:28.588414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588303    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-envoy\") pod \"197109cc-f204-4d13-bf73-02172b3003ec\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") "
Oct 30 21:56:28.588414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588374    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-kube-api-access-sr9k8\" (UniqueName: \"kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-kube-api-access-sr9k8\") pod \"197109cc-f204-4d13-bf73-02172b3003ec\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") "
Oct 30 21:56:28.588723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588425    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-certs\") pod \"197109cc-f204-4d13-bf73-02172b3003ec\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") "
Oct 30 21:56:28.588723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588449    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-data\") pod \"197109cc-f204-4d13-bf73-02172b3003ec\" (UID: \"197109cc-f204-4d13-bf73-02172b3003ec\") "
Oct 30 21:56:28.588723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588613    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-socket" pod "197109cc-f204-4d13-bf73-02172b3003ec" (UID: "197109cc-f204-4d13-bf73-02172b3003ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:28.588723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588624    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-credential-socket" pod "197109cc-f204-4d13-bf73-02172b3003ec" (UID: "197109cc-f204-4d13-bf73-02172b3003ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:28.588723 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588701    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqcfj\" (UniqueName: \"kubernetes.io/projected/b209119c-222d-444f-a597-43fdc39d42ac-kube-api-access-bqcfj\") pod \"maas-api-revoke-sub-379bb5b6-l6d4p\" (UID: \"b209119c-222d-444f-a597-43fdc39d42ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:28.589162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588802    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thz8q\" (UniqueName: \"kubernetes.io/projected/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-kube-api-access-thz8q\") pod \"maas-api-revoke-sub-93653586-m58wl\" (UID: \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:28.589162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588846    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93653586-m58wl\" (UID: \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:28.589162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588927    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-data" pod "197109cc-f204-4d13-bf73-02172b3003ec" (UID: "197109cc-f204-4d13-bf73-02172b3003ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:28.589162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588937    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-379bb5b6-l6d4p\" (UID: \"b209119c-222d-444f-a597-43fdc39d42ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:28.589162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588946    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-certs" pod "197109cc-f204-4d13-bf73-02172b3003ec" (UID: "197109cc-f204-4d13-bf73-02172b3003ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:28.589162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.588991    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-379bb5b6-l6d4p\" (UID: \"b209119c-222d-444f-a597-43fdc39d42ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:28.589162 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.589061    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93653586-m58wl\" (UID: \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:28.589599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.589247    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.589599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.589265    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.589599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.589289    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.589599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.589302    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.591298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.589921    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93653586-m58wl\" (UID: \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:28.591298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.590484    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197109cc-f204-4d13-bf73-02172b3003ec-istiod-ca-cert" pod "197109cc-f204-4d13-bf73-02172b3003ec" (UID: "197109cc-f204-4d13-bf73-02172b3003ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:28.591298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.591168    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-379bb5b6-l6d4p\" (UID: \"b209119c-222d-444f-a597-43fdc39d42ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:28.591298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.591269    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/197109cc-f204-4d13-bf73-02172b3003ec-istio-podinfo" pod "197109cc-f204-4d13-bf73-02172b3003ec" (UID: "197109cc-f204-4d13-bf73-02172b3003ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:28.591677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.591491    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-istio-token" pod "197109cc-f204-4d13-bf73-02172b3003ec" (UID: "197109cc-f204-4d13-bf73-02172b3003ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:28.592157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.592134    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-379bb5b6-l6d4p\" (UID: \"b209119c-222d-444f-a597-43fdc39d42ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:28.592763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.592739    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-envoy" pod "197109cc-f204-4d13-bf73-02172b3003ec" (UID: "197109cc-f204-4d13-bf73-02172b3003ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:28.592907 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.592889    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93653586-m58wl\" (UID: \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:28.593551 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.593529    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-kube-api-access-sr9k8" pod "197109cc-f204-4d13-bf73-02172b3003ec" (UID: "197109cc-f204-4d13-bf73-02172b3003ec"). InnerVolumeSpecName "kube-api-access-sr9k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:28.603316 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.603292    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqcfj\" (UniqueName: \"kubernetes.io/projected/b209119c-222d-444f-a597-43fdc39d42ac-kube-api-access-bqcfj\") pod \"maas-api-revoke-sub-379bb5b6-l6d4p\" (UID: \"b209119c-222d-444f-a597-43fdc39d42ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:28.617615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.617595    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thz8q\" (UniqueName: \"kubernetes.io/projected/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-kube-api-access-thz8q\") pod \"maas-api-revoke-sub-93653586-m58wl\" (UID: \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:28.690461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.690425    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197109cc-f204-4d13-bf73-02172b3003ec-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.690461 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.690455    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197109cc-f204-4d13-bf73-02172b3003ec-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.690672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.690471    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr9k8\" (UniqueName: \"kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-kube-api-access-sr9k8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.690672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.690485    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197109cc-f204-4d13-bf73-02172b3003ec-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.690672 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.690495    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197109cc-f204-4d13-bf73-02172b3003ec-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:28.719534 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.719505    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:28.745918 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:28.745880    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:29.024661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.024630    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"]
Oct 30 21:56:29.026430 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:56:29.026403    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb209119c_222d_444f_a597_43fdc39d42ac.slice/crio-af30fbc35d0d7cbfd1fb048027fec023801b15db1f6969f705fc1d1fc9b5d037 WatchSource:0}: Error finding container af30fbc35d0d7cbfd1fb048027fec023801b15db1f6969f705fc1d1fc9b5d037: Status 404 returned error can't find the container with id af30fbc35d0d7cbfd1fb048027fec023801b15db1f6969f705fc1d1fc9b5d037
Oct 30 21:56:29.071853 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.071827    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"]
Oct 30 21:56:29.072643 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:56:29.072619    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0e04ce_7e47_4e75_afb8_b0c06c7e6cec.slice/crio-940ebfb047b408e77996f15500bba547cba2de2324e088bb7f430a19bd0bc8b8 WatchSource:0}: Error finding container 940ebfb047b408e77996f15500bba547cba2de2324e088bb7f430a19bd0bc8b8: Status 404 returned error can't find the container with id 940ebfb047b408e77996f15500bba547cba2de2324e088bb7f430a19bd0bc8b8
Oct 30 21:56:29.208038 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:29.208002    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:56:29.505467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.505437    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p" event={"ID":"b209119c-222d-444f-a597-43fdc39d42ac","Type":"ContainerStarted","Data":"2a909013c7cc1fca08b0b0e125dcb1cd23aed725d9320dc616c445c5a75c3f95"}
Oct 30 21:56:29.505906 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.505493    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p" event={"ID":"b209119c-222d-444f-a597-43fdc39d42ac","Type":"ContainerStarted","Data":"af30fbc35d0d7cbfd1fb048027fec023801b15db1f6969f705fc1d1fc9b5d037"}
Oct 30 21:56:29.507033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.507013    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"
Oct 30 21:56:29.507168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.507040    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v" event={"ID":"197109cc-f204-4d13-bf73-02172b3003ec","Type":"ContainerDied","Data":"39e7f6ea793dfd2b980e487134be2d241494c33ed76e22b25fb9e86a075a484d"}
Oct 30 21:56:29.507168 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.507092    4918 scope.go:122] "RemoveContainer" containerID="92454911f832d44911996bc3c7e62bc45b068b14d52175e06e70611130a9fff5"
Oct 30 21:56:29.508913 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.508888    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl" event={"ID":"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec","Type":"ContainerStarted","Data":"cc688586e7d84ad7fab5d26b324261a96b6437de5ac74ba9df0808c0fef7dbaf"}
Oct 30 21:56:29.509051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.508931    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl" event={"ID":"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec","Type":"ContainerStarted","Data":"940ebfb047b408e77996f15500bba547cba2de2324e088bb7f430a19bd0bc8b8"}
Oct 30 21:56:29.538042 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.537997    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p" podStartSLOduration=1.5379848109999998 podStartE2EDuration="1.537984811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:29.534709658 +0000 UTC m=+3302.581268820" watchObservedRunningTime="2026-10-30 21:56:29.537984811 +0000 UTC m=+3302.584543955"
Oct 30 21:56:29.569448 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.564803    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl" podStartSLOduration=1.56478775 podStartE2EDuration="1.56478775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:29.560244794 +0000 UTC m=+3302.606803939" watchObservedRunningTime="2026-10-30 21:56:29.56478775 +0000 UTC m=+3302.611346895"
Oct 30 21:56:29.599314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.599281    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"]
Oct 30 21:56:29.620141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:29.620115    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bfc50e-openshift-default-55967d48cc-8tk4v"]
Oct 30 21:56:30.362523 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.362478    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main" containerID="cri-o://ad7a1d0aee6c5115ba59cf87f53146415cbbee5ef314f60fca0a084e82c43226" gracePeriod=45
Oct 30 21:56:30.519973 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.519938    4918 generic.go:356] "Generic (PLEG): container finished" podID="9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec" containerID="cc688586e7d84ad7fab5d26b324261a96b6437de5ac74ba9df0808c0fef7dbaf" exitCode=0
Oct 30 21:56:30.520427 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.520020    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl" event={"ID":"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec","Type":"ContainerDied","Data":"cc688586e7d84ad7fab5d26b324261a96b6437de5ac74ba9df0808c0fef7dbaf"}
Oct 30 21:56:30.521908 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.521885    4918 generic.go:356] "Generic (PLEG): container finished" podID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerID="ad7a1d0aee6c5115ba59cf87f53146415cbbee5ef314f60fca0a084e82c43226" exitCode=0
Oct 30 21:56:30.522002 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.521951    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" event={"ID":"f2e6ce17-cac4-42ab-9a05-d233ea2d9392","Type":"ContainerDied","Data":"ad7a1d0aee6c5115ba59cf87f53146415cbbee5ef314f60fca0a084e82c43226"}
Oct 30 21:56:30.523647 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.523626    4918 generic.go:356] "Generic (PLEG): container finished" podID="b209119c-222d-444f-a597-43fdc39d42ac" containerID="2a909013c7cc1fca08b0b0e125dcb1cd23aed725d9320dc616c445c5a75c3f95" exitCode=0
Oct 30 21:56:30.523755 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.523710    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p" event={"ID":"b209119c-222d-444f-a597-43fdc39d42ac","Type":"ContainerDied","Data":"2a909013c7cc1fca08b0b0e125dcb1cd23aed725d9320dc616c445c5a75c3f95"}
Oct 30 21:56:30.627236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.627179    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:56:30.713795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.713762    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-dshm\") pod \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") "
Oct 30 21:56:30.713795 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.713798    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-home\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-home\") pod \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") "
Oct 30 21:56:30.714230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.713940    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tmp-dir\") pod \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") "
Oct 30 21:56:30.714230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.714046    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-home" pod "f2e6ce17-cac4-42ab-9a05-d233ea2d9392" (UID: "f2e6ce17-cac4-42ab-9a05-d233ea2d9392"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:30.714230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.714058    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tls-certs\") pod \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") "
Oct 30 21:56:30.714230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.714128    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-model-cache\") pod \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") "
Oct 30 21:56:30.714230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.714171    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tmp-dir" pod "f2e6ce17-cac4-42ab-9a05-d233ea2d9392" (UID: "f2e6ce17-cac4-42ab-9a05-d233ea2d9392"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:30.714230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.714204    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-kube-api-access-46mkr\" (UniqueName: \"kubernetes.io/projected/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-kube-api-access-46mkr\") pod \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\" (UID: \"f2e6ce17-cac4-42ab-9a05-d233ea2d9392\") "
Oct 30 21:56:30.714640 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.714367    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-model-cache" pod "f2e6ce17-cac4-42ab-9a05-d233ea2d9392" (UID: "f2e6ce17-cac4-42ab-9a05-d233ea2d9392"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:30.714714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.714661    4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:30.714714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.714680    4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:30.714714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.714695    4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:30.716289 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.716261    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-dshm" pod "f2e6ce17-cac4-42ab-9a05-d233ea2d9392" (UID: "f2e6ce17-cac4-42ab-9a05-d233ea2d9392"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:30.716470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.716299    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tls-certs" pod "f2e6ce17-cac4-42ab-9a05-d233ea2d9392" (UID: "f2e6ce17-cac4-42ab-9a05-d233ea2d9392"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:30.716470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.716447    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-kube-api-access-46mkr" pod "f2e6ce17-cac4-42ab-9a05-d233ea2d9392" (UID: "f2e6ce17-cac4-42ab-9a05-d233ea2d9392"). InnerVolumeSpecName "kube-api-access-46mkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:30.816213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.816180    4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:30.816213 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.816213    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46mkr\" (UniqueName: \"kubernetes.io/projected/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-kube-api-access-46mkr\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:30.816445 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:30.816224    4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e6ce17-cac4-42ab-9a05-d233ea2d9392-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:31.077835 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.077796    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197109cc-f204-4d13-bf73-02172b3003ec" path="/var/lib/kubelet/pods/197109cc-f204-4d13-bf73-02172b3003ec/volumes"
Oct 30 21:56:31.172865 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.172836    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:56:31.174820 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.174793    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:56:31.562263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.562234    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"
Oct 30 21:56:31.570539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.570494    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4" event={"ID":"f2e6ce17-cac4-42ab-9a05-d233ea2d9392","Type":"ContainerDied","Data":"923c4ee4b5f1a0d82606ebd7c4d4d68aaa490195ce79efec07b0a67a2bd612f9"}
Oct 30 21:56:31.570821 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.570792    4918 scope.go:122] "RemoveContainer" containerID="ad7a1d0aee6c5115ba59cf87f53146415cbbee5ef314f60fca0a084e82c43226"
Oct 30 21:56:31.605581 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.605519    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"]
Oct 30 21:56:31.609299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.609272    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a33b12/premium-w1-a33b12-kserve-6db59766f9-72fx4"]
Oct 30 21:56:31.860881 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.860848    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:31.862988 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.862967    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:31.929360 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.929328    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-openshift-ca\") pod \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\" (UID: \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\") "
Oct 30 21:56:31.929580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.929471    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-xks-ca\") pod \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\" (UID: \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\") "
Oct 30 21:56:31.929580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.929537    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b209119c-222d-444f-a597-43fdc39d42ac-kube-api-access-bqcfj\" (UniqueName: \"kubernetes.io/projected/b209119c-222d-444f-a597-43fdc39d42ac-kube-api-access-bqcfj\") pod \"b209119c-222d-444f-a597-43fdc39d42ac\" (UID: \"b209119c-222d-444f-a597-43fdc39d42ac\") "
Oct 30 21:56:31.929863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.929601    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-kube-api-access-thz8q\" (UniqueName: \"kubernetes.io/projected/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-kube-api-access-thz8q\") pod \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\" (UID: \"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec\") "
Oct 30 21:56:31.929863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.929645    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-xks-ca\") pod \"b209119c-222d-444f-a597-43fdc39d42ac\" (UID: \"b209119c-222d-444f-a597-43fdc39d42ac\") "
Oct 30 21:56:31.929863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.929703    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-openshift-ca\") pod \"b209119c-222d-444f-a597-43fdc39d42ac\" (UID: \"b209119c-222d-444f-a597-43fdc39d42ac\") "
Oct 30 21:56:31.929863 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.929756    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-openshift-ca" pod "9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec" (UID: "9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:31.930181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.930122    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:31.930574 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.930544    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-openshift-ca" pod "b209119c-222d-444f-a597-43fdc39d42ac" (UID: "b209119c-222d-444f-a597-43fdc39d42ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:31.933549 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.933524    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-xks-ca" pod "b209119c-222d-444f-a597-43fdc39d42ac" (UID: "b209119c-222d-444f-a597-43fdc39d42ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:31.933654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.933545    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-xks-ca" pod "9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec" (UID: "9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:31.933654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.933561    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-kube-api-access-thz8q" pod "9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec" (UID: "9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec"). InnerVolumeSpecName "kube-api-access-thz8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:31.933654 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:31.933587    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b209119c-222d-444f-a597-43fdc39d42ac-kube-api-access-bqcfj" pod "b209119c-222d-444f-a597-43fdc39d42ac" (UID: "b209119c-222d-444f-a597-43fdc39d42ac"). InnerVolumeSpecName "kube-api-access-bqcfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:32.031482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.031453    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:32.031482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.031480    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:32.031696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.031492    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqcfj\" (UniqueName: \"kubernetes.io/projected/b209119c-222d-444f-a597-43fdc39d42ac-kube-api-access-bqcfj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:32.031696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.031503    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thz8q\" (UniqueName: \"kubernetes.io/projected/9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec-kube-api-access-thz8q\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:32.031696 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.031513    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b209119c-222d-444f-a597-43fdc39d42ac-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:32.429650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.429610    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"]
Oct 30 21:56:32.429887 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.429862    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy" containerID="cri-o://d633a872e2055586db2c1ec84f203ae6ff8cce330bed76aa9569cf9b66765834" gracePeriod=30
Oct 30 21:56:32.438583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.438547    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"]
Oct 30 21:56:32.438909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.438883    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds" podUID="a16b9f2a-0ed2-41ce-b956-a5d5d0081788" containerName="istio-proxy" containerID="cri-o://75839efcc128588612537b2a6031e36f90a132a2e08c88d857c80879f934be0b" gracePeriod=30
Oct 30 21:56:32.575207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.575182    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl"
Oct 30 21:56:32.575756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.575204    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93653586-m58wl" event={"ID":"9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec","Type":"ContainerDied","Data":"940ebfb047b408e77996f15500bba547cba2de2324e088bb7f430a19bd0bc8b8"}
Oct 30 21:56:32.575756 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.575264    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="940ebfb047b408e77996f15500bba547cba2de2324e088bb7f430a19bd0bc8b8"
Oct 30 21:56:32.578497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.578472    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p"
Oct 30 21:56:32.578644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.578505    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-379bb5b6-l6d4p" event={"ID":"b209119c-222d-444f-a597-43fdc39d42ac","Type":"ContainerDied","Data":"af30fbc35d0d7cbfd1fb048027fec023801b15db1f6969f705fc1d1fc9b5d037"}
Oct 30 21:56:32.578644 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.578542    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af30fbc35d0d7cbfd1fb048027fec023801b15db1f6969f705fc1d1fc9b5d037"
Oct 30 21:56:32.968735 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.968702    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"]
Oct 30 21:56:32.969151 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.969138    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec" containerName="revoke-keys"
Oct 30 21:56:32.969202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.969152    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec" containerName="revoke-keys"
Oct 30 21:56:32.969202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.969174    4918 cpu_manager.go:414] "containerMap: removing container" podUID="b209119c-222d-444f-a597-43fdc39d42ac" containerName="revoke-keys"
Oct 30 21:56:32.969202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.969182    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b209119c-222d-444f-a597-43fdc39d42ac" containerName="revoke-keys"
Oct 30 21:56:32.969297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.969209    4918 cpu_manager.go:414] "containerMap: removing container" podUID="197109cc-f204-4d13-bf73-02172b3003ec" containerName="istio-proxy"
Oct 30 21:56:32.969297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.969214    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197109cc-f204-4d13-bf73-02172b3003ec" containerName="istio-proxy"
Oct 30 21:56:32.969297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.969227    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main"
Oct 30 21:56:32.969297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.969231    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main"
Oct 30 21:56:32.969468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.969302    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec" containerName="revoke-keys"
Oct 30 21:56:32.969468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.969314    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="b209119c-222d-444f-a597-43fdc39d42ac" containerName="revoke-keys"
Oct 30 21:56:32.969468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.969323    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="197109cc-f204-4d13-bf73-02172b3003ec" containerName="istio-proxy"
Oct 30 21:56:32.969468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.969329    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" containerName="main"
Oct 30 21:56:32.973715 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.973697    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:32.978888 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.978868    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:32.979048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.979016    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:56:32.979441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.979423    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:56:32.979441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.979422    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:56:32.988049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:32.988028    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"]
Oct 30 21:56:33.049361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:33.049327    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjq2w\" (UniqueName: \"kubernetes.io/projected/ac6c6737-7897-40ba-ba36-6bbcddf7496e-kube-api-access-fjq2w\") pod \"maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c\" (UID: \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:33.049561 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:33.049383    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c\" (UID: \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:33.049614 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:33.049553    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c\" (UID: \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:33.076252 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:33.076226    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e6ce17-cac4-42ab-9a05-d233ea2d9392" path="/var/lib/kubelet/pods/f2e6ce17-cac4-42ab-9a05-d233ea2d9392/volumes"
Oct 30 21:56:33.150165 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:33.150136    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c\" (UID: \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:33.150339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:33.150230    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjq2w\" (UniqueName: \"kubernetes.io/projected/ac6c6737-7897-40ba-ba36-6bbcddf7496e-kube-api-access-fjq2w\") pod \"maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c\" (UID: \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:33.150339 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:33.150271    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c\" (UID: \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:33.150852 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:33.150825    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c\" (UID: \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:33.152713 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:33.152695    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c\" (UID: \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:33.162255 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:33.162230    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjq2w\" (UniqueName: \"kubernetes.io/projected/ac6c6737-7897-40ba-ba36-6bbcddf7496e-kube-api-access-fjq2w\") pod \"maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c\" (UID: \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:33.287981 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:33.287950    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:33.579707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:33.579672    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"]
Oct 30 21:56:33.580149 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:56:33.580030    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6c6737_7897_40ba_ba36_6bbcddf7496e.slice/crio-012f1aeb59b317c1e759a1b6c413fc35a964eb302fd8e25c7bc437170788e214 WatchSource:0}: Error finding container 012f1aeb59b317c1e759a1b6c413fc35a964eb302fd8e25c7bc437170788e214: Status 404 returned error can't find the container with id 012f1aeb59b317c1e759a1b6c413fc35a964eb302fd8e25c7bc437170788e214
Oct 30 21:56:34.603100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:34.603069    4918 generic.go:356] "Generic (PLEG): container finished" podID="ac6c6737-7897-40ba-ba36-6bbcddf7496e" containerID="f2d81313fcc38c353cf68813cd1e9aaecdc90bb319ebeb2e66e131fa0dc3ec0f" exitCode=0
Oct 30 21:56:34.603510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:34.603157    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c" event={"ID":"ac6c6737-7897-40ba-ba36-6bbcddf7496e","Type":"ContainerDied","Data":"f2d81313fcc38c353cf68813cd1e9aaecdc90bb319ebeb2e66e131fa0dc3ec0f"}
Oct 30 21:56:34.603510 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:34.603208    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c" event={"ID":"ac6c6737-7897-40ba-ba36-6bbcddf7496e","Type":"ContainerStarted","Data":"012f1aeb59b317c1e759a1b6c413fc35a964eb302fd8e25c7bc437170788e214"}
Oct 30 21:56:34.936357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:34.936284    4918 patch_prober.go:29] interesting pod/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:56:34.936357 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:34.936333    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:56:35.413653 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.413625    4918 scope.go:122] "RemoveContainer" containerID="c4f354300479e1180fac0f1195f25b0c107b127e9265702a990ba92dba149be0"
Oct 30 21:56:35.428508 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.428484    4918 scope.go:122] "RemoveContainer" containerID="166bc2dce5cb3cb357bd4566b40857caf66c6524f95beae268835119286a0be0"
Oct 30 21:56:35.475317 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.475294    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"]
Oct 30 21:56:35.475547 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.475524    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" podUID="71dbe041-9629-4ee0-8221-79c215ac06d8" containerName="istio-proxy" containerID="cri-o://fbbcaa1419f9dc011f5101982a6e6d57d236cd603d26f4e845e8fc4663bce946" gracePeriod=30
Oct 30 21:56:35.756998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.756973    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:35.783718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.783684    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-xks-ca\") pod \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\" (UID: \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\") "
Oct 30 21:56:35.783862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.783824    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6c6737-7897-40ba-ba36-6bbcddf7496e-kube-api-access-fjq2w\" (UniqueName: \"kubernetes.io/projected/ac6c6737-7897-40ba-ba36-6bbcddf7496e-kube-api-access-fjq2w\") pod \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\" (UID: \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\") "
Oct 30 21:56:35.783862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.783852    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-openshift-ca\") pod \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\" (UID: \"ac6c6737-7897-40ba-ba36-6bbcddf7496e\") "
Oct 30 21:56:35.784365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.784338    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-openshift-ca" pod "ac6c6737-7897-40ba-ba36-6bbcddf7496e" (UID: "ac6c6737-7897-40ba-ba36-6bbcddf7496e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:35.784532 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.784515    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:35.786049 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.786023    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-xks-ca" pod "ac6c6737-7897-40ba-ba36-6bbcddf7496e" (UID: "ac6c6737-7897-40ba-ba36-6bbcddf7496e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:35.786170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.786152    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6c6737-7897-40ba-ba36-6bbcddf7496e-kube-api-access-fjq2w" pod "ac6c6737-7897-40ba-ba36-6bbcddf7496e" (UID: "ac6c6737-7897-40ba-ba36-6bbcddf7496e"). InnerVolumeSpecName "kube-api-access-fjq2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:35.886045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.886003    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6c6737-7897-40ba-ba36-6bbcddf7496e-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:35.886045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:35.886038    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjq2w\" (UniqueName: \"kubernetes.io/projected/ac6c6737-7897-40ba-ba36-6bbcddf7496e-kube-api-access-fjq2w\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:36.075450 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:36.075381    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm" podUID="ff7f4bf0-e541-4893-a000-027b14e42b74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:36.075618 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:36.075423    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7" podUID="a13a507e-48ca-4b4a-90dd-72f30316c75d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:36.619878 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:36.619840    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"
Oct 30 21:56:36.620066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:36.619875    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c" event={"ID":"ac6c6737-7897-40ba-ba36-6bbcddf7496e","Type":"ContainerDied","Data":"012f1aeb59b317c1e759a1b6c413fc35a964eb302fd8e25c7bc437170788e214"}
Oct 30 21:56:36.620066 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:36.619960    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="012f1aeb59b317c1e759a1b6c413fc35a964eb302fd8e25c7bc437170788e214"
Oct 30 21:56:37.631276 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.631240    4918 generic.go:356] "Generic (PLEG): container finished" podID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerID="d633a872e2055586db2c1ec84f203ae6ff8cce330bed76aa9569cf9b66765834" exitCode=0
Oct 30 21:56:37.631749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.631376    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" event={"ID":"fc03a02c-5042-4f3c-be33-4443682c2f6d","Type":"ContainerDied","Data":"d633a872e2055586db2c1ec84f203ae6ff8cce330bed76aa9569cf9b66765834"}
Oct 30 21:56:37.631749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.631458    4918 scope.go:122] "RemoveContainer" containerID="a5f85956238ef40fa02cf7d14786d6184a719746e22d129a6dc239b3bed068a4"
Oct 30 21:56:37.634942 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.634913    4918 generic.go:356] "Generic (PLEG): container finished" podID="a16b9f2a-0ed2-41ce-b956-a5d5d0081788" containerID="75839efcc128588612537b2a6031e36f90a132a2e08c88d857c80879f934be0b" exitCode=0
Oct 30 21:56:37.635052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.634973    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds" event={"ID":"a16b9f2a-0ed2-41ce-b956-a5d5d0081788","Type":"ContainerDied","Data":"75839efcc128588612537b2a6031e36f90a132a2e08c88d857c80879f934be0b"}
Oct 30 21:56:37.704569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.704516    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:56:37.712264 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.712229    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-podinfo\") pod \"fc03a02c-5042-4f3c-be33-4443682c2f6d\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") "
Oct 30 21:56:37.712500 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.712478    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc03a02c-5042-4f3c-be33-4443682c2f6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc03a02c-5042-4f3c-be33-4443682c2f6d-istiod-ca-cert\") pod \"fc03a02c-5042-4f3c-be33-4443682c2f6d\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") "
Oct 30 21:56:37.712592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.712575    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-socket\") pod \"fc03a02c-5042-4f3c-be33-4443682c2f6d\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") "
Oct 30 21:56:37.712662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.712626    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-credential-socket\") pod \"fc03a02c-5042-4f3c-be33-4443682c2f6d\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") "
Oct 30 21:56:37.712748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.712659    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-data\") pod \"fc03a02c-5042-4f3c-be33-4443682c2f6d\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") "
Oct 30 21:56:37.712748 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.712701    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-token\" (UniqueName: \"kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-token\") pod \"fc03a02c-5042-4f3c-be33-4443682c2f6d\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") "
Oct 30 21:56:37.712891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.712869    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-socket" pod "fc03a02c-5042-4f3c-be33-4443682c2f6d" (UID: "fc03a02c-5042-4f3c-be33-4443682c2f6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.712891 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.712875    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc03a02c-5042-4f3c-be33-4443682c2f6d-istiod-ca-cert" pod "fc03a02c-5042-4f3c-be33-4443682c2f6d" (UID: "fc03a02c-5042-4f3c-be33-4443682c2f6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:37.713109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.713083    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-data" pod "fc03a02c-5042-4f3c-be33-4443682c2f6d" (UID: "fc03a02c-5042-4f3c-be33-4443682c2f6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.713237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.713219    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc03a02c-5042-4f3c-be33-4443682c2f6d-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.713330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.713245    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.713330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.713260    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.713506 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.713384    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-credential-socket" pod "fc03a02c-5042-4f3c-be33-4443682c2f6d" (UID: "fc03a02c-5042-4f3c-be33-4443682c2f6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.714902 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.714878    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-podinfo" pod "fc03a02c-5042-4f3c-be33-4443682c2f6d" (UID: "fc03a02c-5042-4f3c-be33-4443682c2f6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:37.715092 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.715075    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-token" pod "fc03a02c-5042-4f3c-be33-4443682c2f6d" (UID: "fc03a02c-5042-4f3c-be33-4443682c2f6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:37.733263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.733211    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:56:37.813737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.813704    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-certs\") pod \"fc03a02c-5042-4f3c-be33-4443682c2f6d\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") "
Oct 30 21:56:37.813737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.813741    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-envoy\") pod \"fc03a02c-5042-4f3c-be33-4443682c2f6d\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") "
Oct 30 21:56:37.813995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.813773    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-credential-socket\") pod \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") "
Oct 30 21:56:37.813995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.813793    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-kube-api-access-f8kl4\" (UniqueName: \"kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-kube-api-access-f8kl4\") pod \"fc03a02c-5042-4f3c-be33-4443682c2f6d\" (UID: \"fc03a02c-5042-4f3c-be33-4443682c2f6d\") "
Oct 30 21:56:37.813995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.813821    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-data\") pod \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") "
Oct 30 21:56:37.813995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.813848    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istiod-ca-cert\") pod \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") "
Oct 30 21:56:37.813995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.813882    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-podinfo\") pod \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") "
Oct 30 21:56:37.813995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.813900    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-envoy\") pod \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") "
Oct 30 21:56:37.813995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.813936    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-token\" (UniqueName: \"kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-token\") pod \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") "
Oct 30 21:56:37.813995 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.813965    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-socket\") pod \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") "
Oct 30 21:56:37.814474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814000    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-kube-api-access-wq9cd\" (UniqueName: \"kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-kube-api-access-wq9cd\") pod \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") "
Oct 30 21:56:37.814474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814035    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-certs" pod "fc03a02c-5042-4f3c-be33-4443682c2f6d" (UID: "fc03a02c-5042-4f3c-be33-4443682c2f6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.814474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814064    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-certs\") pod \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\" (UID: \"a16b9f2a-0ed2-41ce-b956-a5d5d0081788\") "
Oct 30 21:56:37.814474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814269    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-data" pod "a16b9f2a-0ed2-41ce-b956-a5d5d0081788" (UID: "a16b9f2a-0ed2-41ce-b956-a5d5d0081788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.814474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814348    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-socket" pod "a16b9f2a-0ed2-41ce-b956-a5d5d0081788" (UID: "a16b9f2a-0ed2-41ce-b956-a5d5d0081788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.814474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814424    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-credential-socket" pod "a16b9f2a-0ed2-41ce-b956-a5d5d0081788" (UID: "a16b9f2a-0ed2-41ce-b956-a5d5d0081788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.814998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814557    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.814998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814578    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.814998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814593    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.814998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814607    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.814998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814620    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.814998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814633    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.814998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814677    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.814998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.814831    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istiod-ca-cert" pod "a16b9f2a-0ed2-41ce-b956-a5d5d0081788" (UID: "a16b9f2a-0ed2-41ce-b956-a5d5d0081788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:37.815470 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.815266    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-certs" pod "a16b9f2a-0ed2-41ce-b956-a5d5d0081788" (UID: "a16b9f2a-0ed2-41ce-b956-a5d5d0081788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.816750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.816721    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-kube-api-access-f8kl4" pod "fc03a02c-5042-4f3c-be33-4443682c2f6d" (UID: "fc03a02c-5042-4f3c-be33-4443682c2f6d"). InnerVolumeSpecName "kube-api-access-f8kl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:37.816975 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.816949    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-podinfo" pod "a16b9f2a-0ed2-41ce-b956-a5d5d0081788" (UID: "a16b9f2a-0ed2-41ce-b956-a5d5d0081788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:37.817137 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.817118    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-envoy" pod "fc03a02c-5042-4f3c-be33-4443682c2f6d" (UID: "fc03a02c-5042-4f3c-be33-4443682c2f6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.817361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.817340    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-token" pod "a16b9f2a-0ed2-41ce-b956-a5d5d0081788" (UID: "a16b9f2a-0ed2-41ce-b956-a5d5d0081788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:37.817504 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.817486    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-envoy" pod "a16b9f2a-0ed2-41ce-b956-a5d5d0081788" (UID: "a16b9f2a-0ed2-41ce-b956-a5d5d0081788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:37.817585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.817545    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-kube-api-access-wq9cd" pod "a16b9f2a-0ed2-41ce-b956-a5d5d0081788" (UID: "a16b9f2a-0ed2-41ce-b956-a5d5d0081788"). InnerVolumeSpecName "kube-api-access-wq9cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:37.915545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.915519    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.915545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.915543    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq9cd\" (UniqueName: \"kubernetes.io/projected/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-kube-api-access-wq9cd\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.915749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.915553    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.915749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.915562    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc03a02c-5042-4f3c-be33-4443682c2f6d-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.915749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.915571    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8kl4\" (UniqueName: \"kubernetes.io/projected/fc03a02c-5042-4f3c-be33-4443682c2f6d-kube-api-access-f8kl4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.915749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.915578    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.915749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.915586    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.915749 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:37.915594    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a16b9f2a-0ed2-41ce-b956-a5d5d0081788-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:38.212102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:38.212062    4918 patch_prober.go:29] interesting pod/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:56:38.212272 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:38.212123    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" podUID="71dbe041-9629-4ee0-8221-79c215ac06d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:56:38.644657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:38.644626    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"
Oct 30 21:56:38.644657 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:38.644645    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc" event={"ID":"fc03a02c-5042-4f3c-be33-4443682c2f6d","Type":"ContainerDied","Data":"fd2b6fa9321598741436afb10461eab82e183b7184354685ec2344a409501fc1"}
Oct 30 21:56:38.645187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:38.644697    4918 scope.go:122] "RemoveContainer" containerID="d633a872e2055586db2c1ec84f203ae6ff8cce330bed76aa9569cf9b66765834"
Oct 30 21:56:38.646236 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:38.646220    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"
Oct 30 21:56:38.646369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:38.646341    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds" event={"ID":"a16b9f2a-0ed2-41ce-b956-a5d5d0081788","Type":"ContainerDied","Data":"f896020e9b8979a5d095c9d98c7ce945a1aa65cd6939a43823f9d46bb399df3a"}
Oct 30 21:56:38.655934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:38.655917    4918 scope.go:122] "RemoveContainer" containerID="75839efcc128588612537b2a6031e36f90a132a2e08c88d857c80879f934be0b"
Oct 30 21:56:38.685020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:38.684997    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"]
Oct 30 21:56:38.697867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:38.697844    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-4nlhc"]
Oct 30 21:56:38.715970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:38.715947    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"]
Oct 30 21:56:38.725864 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:38.725842    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-fb2b58-openshift-default-66696d54d9-r85ds"]
Oct 30 21:56:39.077871 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:39.077839    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a16b9f2a-0ed2-41ce-b956-a5d5d0081788" path="/var/lib/kubelet/pods/a16b9f2a-0ed2-41ce-b956-a5d5d0081788/volumes"
Oct 30 21:56:39.078334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:39.078313    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" path="/var/lib/kubelet/pods/fc03a02c-5042-4f3c-be33-4443682c2f6d/volumes"
Oct 30 21:56:40.664441 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.664406    4918 generic.go:356] "Generic (PLEG): container finished" podID="71dbe041-9629-4ee0-8221-79c215ac06d8" containerID="fbbcaa1419f9dc011f5101982a6e6d57d236cd603d26f4e845e8fc4663bce946" exitCode=0
Oct 30 21:56:40.664931 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.664468    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" event={"ID":"71dbe041-9629-4ee0-8221-79c215ac06d8","Type":"ContainerDied","Data":"fbbcaa1419f9dc011f5101982a6e6d57d236cd603d26f4e845e8fc4663bce946"}
Oct 30 21:56:40.723940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.723915    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:56:40.742334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742281    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-certs\") pod \"71dbe041-9629-4ee0-8221-79c215ac06d8\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") "
Oct 30 21:56:40.742334 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742313    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-data\") pod \"71dbe041-9629-4ee0-8221-79c215ac06d8\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") "
Oct 30 21:56:40.742559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742351    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-socket\") pod \"71dbe041-9629-4ee0-8221-79c215ac06d8\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") "
Oct 30 21:56:40.742559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742435    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-kube-api-access-fk6t2\" (UniqueName: \"kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-kube-api-access-fk6t2\") pod \"71dbe041-9629-4ee0-8221-79c215ac06d8\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") "
Oct 30 21:56:40.742559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742465    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-envoy\") pod \"71dbe041-9629-4ee0-8221-79c215ac06d8\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") "
Oct 30 21:56:40.742559 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742522    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-podinfo\") pod \"71dbe041-9629-4ee0-8221-79c215ac06d8\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") "
Oct 30 21:56:40.742804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742607    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-credential-socket\") pod \"71dbe041-9629-4ee0-8221-79c215ac06d8\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") "
Oct 30 21:56:40.742804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742610    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-certs" pod "71dbe041-9629-4ee0-8221-79c215ac06d8" (UID: "71dbe041-9629-4ee0-8221-79c215ac06d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:40.742804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742629    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-data" pod "71dbe041-9629-4ee0-8221-79c215ac06d8" (UID: "71dbe041-9629-4ee0-8221-79c215ac06d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:40.742804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742655    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71dbe041-9629-4ee0-8221-79c215ac06d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dbe041-9629-4ee0-8221-79c215ac06d8-istiod-ca-cert\") pod \"71dbe041-9629-4ee0-8221-79c215ac06d8\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") "
Oct 30 21:56:40.742804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742718    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-socket" pod "71dbe041-9629-4ee0-8221-79c215ac06d8" (UID: "71dbe041-9629-4ee0-8221-79c215ac06d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:40.742804 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742722    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-token\" (UniqueName: \"kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-token\") pod \"71dbe041-9629-4ee0-8221-79c215ac06d8\" (UID: \"71dbe041-9629-4ee0-8221-79c215ac06d8\") "
Oct 30 21:56:40.743159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.742893    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-credential-socket" pod "71dbe041-9629-4ee0-8221-79c215ac06d8" (UID: "71dbe041-9629-4ee0-8221-79c215ac06d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:40.743159 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.743034    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71dbe041-9629-4ee0-8221-79c215ac06d8-istiod-ca-cert" pod "71dbe041-9629-4ee0-8221-79c215ac06d8" (UID: "71dbe041-9629-4ee0-8221-79c215ac06d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:40.743474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.743233    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71dbe041-9629-4ee0-8221-79c215ac06d8-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.743474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.743255    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.743474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.743316    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.743474 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.743471    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.743722 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.743487    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.744837 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.744809    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-podinfo" pod "71dbe041-9629-4ee0-8221-79c215ac06d8" (UID: "71dbe041-9629-4ee0-8221-79c215ac06d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:56:40.744972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.744952    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-envoy" pod "71dbe041-9629-4ee0-8221-79c215ac06d8" (UID: "71dbe041-9629-4ee0-8221-79c215ac06d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:56:40.744972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.744962    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-kube-api-access-fk6t2" pod "71dbe041-9629-4ee0-8221-79c215ac06d8" (UID: "71dbe041-9629-4ee0-8221-79c215ac06d8"). InnerVolumeSpecName "kube-api-access-fk6t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:40.745101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.745051    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-token" pod "71dbe041-9629-4ee0-8221-79c215ac06d8" (UID: "71dbe041-9629-4ee0-8221-79c215ac06d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:40.844797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.844763    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.844797 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.844793    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.845006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.844807    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk6t2\" (UniqueName: \"kubernetes.io/projected/71dbe041-9629-4ee0-8221-79c215ac06d8-kube-api-access-fk6t2\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:40.845006 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:40.844820    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71dbe041-9629-4ee0-8221-79c215ac06d8-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:41.671904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:41.671876    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"
Oct 30 21:56:41.672355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:41.671877    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q" event={"ID":"71dbe041-9629-4ee0-8221-79c215ac06d8","Type":"ContainerDied","Data":"ad9b442b742eabac582e53a2e0927fb9e5d892a65dea5a5a7a3f078e10f88ceb"}
Oct 30 21:56:41.672355 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:41.672017    4918 scope.go:122] "RemoveContainer" containerID="fbbcaa1419f9dc011f5101982a6e6d57d236cd603d26f4e845e8fc4663bce946"
Oct 30 21:56:41.694751 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:41.694728    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"]
Oct 30 21:56:41.704295 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:41.704273    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3011c6-openshift-default-78b56fc845-9gp8q"]
Oct 30 21:56:43.076516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:43.076485    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71dbe041-9629-4ee0-8221-79c215ac06d8" path="/var/lib/kubelet/pods/71dbe041-9629-4ee0-8221-79c215ac06d8/volumes"
Oct 30 21:56:43.253142 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:43.253105    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"]
Oct 30 21:56:46.074360 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:56:46.074328    4918 logging.go:55] [core] [Channel #15595 SubChannel #15596]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.57:50052", ServerName: "10.132.0.57:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.57:50052: connect: connection refused"
Oct 30 21:56:46.135214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.135191    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"
Oct 30 21:56:46.175777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.175743    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"
Oct 30 21:56:46.206720 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.206690    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff7f4bf0-e541-4893-a000-027b14e42b74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7f4bf0-e541-4893-a000-027b14e42b74-plugins-config-volume\") pod \"ff7f4bf0-e541-4893-a000-027b14e42b74\" (UID: \"ff7f4bf0-e541-4893-a000-027b14e42b74\") "
Oct 30 21:56:46.206925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.206763    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13a507e-48ca-4b4a-90dd-72f30316c75d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13a507e-48ca-4b4a-90dd-72f30316c75d-plugins-config-volume\") pod \"a13a507e-48ca-4b4a-90dd-72f30316c75d\" (UID: \"a13a507e-48ca-4b4a-90dd-72f30316c75d\") "
Oct 30 21:56:46.206925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.206788    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13a507e-48ca-4b4a-90dd-72f30316c75d-kube-api-access-hnbtg\" (UniqueName: \"kubernetes.io/projected/a13a507e-48ca-4b4a-90dd-72f30316c75d-kube-api-access-hnbtg\") pod \"a13a507e-48ca-4b4a-90dd-72f30316c75d\" (UID: \"a13a507e-48ca-4b4a-90dd-72f30316c75d\") "
Oct 30 21:56:46.206925 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.206882    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff7f4bf0-e541-4893-a000-027b14e42b74-kube-api-access-5ndjj\" (UniqueName: \"kubernetes.io/projected/ff7f4bf0-e541-4893-a000-027b14e42b74-kube-api-access-5ndjj\") pod \"ff7f4bf0-e541-4893-a000-027b14e42b74\" (UID: \"ff7f4bf0-e541-4893-a000-027b14e42b74\") "
Oct 30 21:56:46.207839 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.207807    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff7f4bf0-e541-4893-a000-027b14e42b74-plugins-config-volume" pod "ff7f4bf0-e541-4893-a000-027b14e42b74" (UID: "ff7f4bf0-e541-4893-a000-027b14e42b74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:46.207970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.207909    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13a507e-48ca-4b4a-90dd-72f30316c75d-plugins-config-volume" pod "a13a507e-48ca-4b4a-90dd-72f30316c75d" (UID: "a13a507e-48ca-4b4a-90dd-72f30316c75d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:46.209834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.209802    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13a507e-48ca-4b4a-90dd-72f30316c75d-kube-api-access-hnbtg" pod "a13a507e-48ca-4b4a-90dd-72f30316c75d" (UID: "a13a507e-48ca-4b4a-90dd-72f30316c75d"). InnerVolumeSpecName "kube-api-access-hnbtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:46.210383 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.210362    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7f4bf0-e541-4893-a000-027b14e42b74-kube-api-access-5ndjj" pod "ff7f4bf0-e541-4893-a000-027b14e42b74" (UID: "ff7f4bf0-e541-4893-a000-027b14e42b74"). InnerVolumeSpecName "kube-api-access-5ndjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:46.308465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.308423    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ndjj\" (UniqueName: \"kubernetes.io/projected/ff7f4bf0-e541-4893-a000-027b14e42b74-kube-api-access-5ndjj\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:46.308465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.308451    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff7f4bf0-e541-4893-a000-027b14e42b74-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:46.308465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.308461    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a13a507e-48ca-4b4a-90dd-72f30316c75d-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:46.308465 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.308470    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnbtg\" (UniqueName: \"kubernetes.io/projected/a13a507e-48ca-4b4a-90dd-72f30316c75d-kube-api-access-hnbtg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:46.709111 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.709076    4918 generic.go:356] "Generic (PLEG): container finished" podID="ff7f4bf0-e541-4893-a000-027b14e42b74" containerID="bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c" exitCode=0
Oct 30 21:56:46.709299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.709123    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"
Oct 30 21:56:46.709299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.709157    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm" event={"ID":"ff7f4bf0-e541-4893-a000-027b14e42b74","Type":"ContainerDied","Data":"bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c"}
Oct 30 21:56:46.709299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.709205    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm" event={"ID":"ff7f4bf0-e541-4893-a000-027b14e42b74","Type":"ContainerDied","Data":"b51511454e9522e3b0b26f6b9ccfcdfbdb36d7b4d0aa68da9b6616d77d3b43b7"}
Oct 30 21:56:46.709299 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.709230    4918 scope.go:122] "RemoveContainer" containerID="bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c"
Oct 30 21:56:46.710801 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.710771    4918 generic.go:356] "Generic (PLEG): container finished" podID="a13a507e-48ca-4b4a-90dd-72f30316c75d" containerID="92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342" exitCode=0
Oct 30 21:56:46.710922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.710817    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"
Oct 30 21:56:46.710922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.710831    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7" event={"ID":"a13a507e-48ca-4b4a-90dd-72f30316c75d","Type":"ContainerDied","Data":"92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342"}
Oct 30 21:56:46.710922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.710876    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7" event={"ID":"a13a507e-48ca-4b4a-90dd-72f30316c75d","Type":"ContainerDied","Data":"6c81488c0d01a980840c4f4f0c3dc4a9df2b6574123d7572ba9c7702779cac0e"}
Oct 30 21:56:46.721738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.721696    4918 scope.go:122] "RemoveContainer" containerID="bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c"
Oct 30 21:56:46.722163 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:46.722041    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c\": container with ID starting with bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c not found: ID does not exist" containerID="bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c"
Oct 30 21:56:46.722163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.722074    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c"} err="failed to get container status \"bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c\": rpc error: code = NotFound desc = could not find container \"bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c\": container with ID starting with bc5163705c00d0e13076171cae792a9c5bd8e404edf60ff2fd7d215743d4b67c not found: ID does not exist"
Oct 30 21:56:46.722163 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.722095    4918 scope.go:122] "RemoveContainer" containerID="92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342"
Oct 30 21:56:46.731058 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.731042    4918 scope.go:122] "RemoveContainer" containerID="92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342"
Oct 30 21:56:46.731335 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:46.731318    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342\": container with ID starting with 92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342 not found: ID does not exist" containerID="92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342"
Oct 30 21:56:46.731416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.731341    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342"} err="failed to get container status \"92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342\": rpc error: code = NotFound desc = could not find container \"92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342\": container with ID starting with 92ff29fa9af4164420dcf5567080ba4a14baa423f088170ddc530e3e86b0f342 not found: ID does not exist"
Oct 30 21:56:46.771116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.771084    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"]
Oct 30 21:56:46.795075 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.795047    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7"]
Oct 30 21:56:46.824789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.824763    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"]
Oct 30 21:56:46.848109 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:46.848085    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm"]
Oct 30 21:56:47.075041 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:47.074998    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a33b12-6cc74cf6f7-nnqtm" podUID="ff7f4bf0-e541-4893-a000-027b14e42b74" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.58:50052\" within 1s: context deadline exceeded"
Oct 30 21:56:47.075560 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:47.074999    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a33b12-6f55dc5bf9-cnll7" podUID="a13a507e-48ca-4b4a-90dd-72f30316c75d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.57:50052\" within 1s: context deadline exceeded"
Oct 30 21:56:47.075560 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:56:47.075080    4918 logging.go:55] [core] [Channel #15594 SubChannel #15597]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.58:50052", ServerName: "10.132.0.58:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.58:50052: operation was canceled"
Oct 30 21:56:47.077929 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:47.077905    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13a507e-48ca-4b4a-90dd-72f30316c75d" path="/var/lib/kubelet/pods/a13a507e-48ca-4b4a-90dd-72f30316c75d/volumes"
Oct 30 21:56:47.078311 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:47.078298    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff7f4bf0-e541-4893-a000-027b14e42b74" path="/var/lib/kubelet/pods/ff7f4bf0-e541-4893-a000-027b14e42b74/volumes"
Oct 30 21:56:47.913024 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:47.912983    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"]
Oct 30 21:56:47.917772 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:47.917744    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0ea1ea-z7p4c"]
Oct 30 21:56:48.253996 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:48.253954    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb" podUID="836991d9-b095-44e2-83af-b8a260eb2413" containerName="payload-processing" containerID="cri-o://ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650" gracePeriod=25
Oct 30 21:56:48.352596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:48.352565    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"]
Oct 30 21:56:48.362859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:48.362830    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a33b12-jhm6b"]
Oct 30 21:56:49.076101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:49.076065    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42fb6e0e-26c4-4425-b604-fd43bf15f226" path="/var/lib/kubelet/pods/42fb6e0e-26c4-4425-b604-fd43bf15f226/volumes"
Oct 30 21:56:49.076530 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:49.076514    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6c6737-7897-40ba-ba36-6bbcddf7496e" path="/var/lib/kubelet/pods/ac6c6737-7897-40ba-ba36-6bbcddf7496e/volumes"
Oct 30 21:56:54.856442 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:56:54.856379    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb" podUID="836991d9-b095-44e2-83af-b8a260eb2413" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:56:59.159827 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:56:59.159792    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:57:00.533916 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:00.533872    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"]
Oct 30 21:57:00.534484 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:00.534129    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" podUID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" containerName="istio-proxy" containerID="cri-o://4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37" gracePeriod=30
Oct 30 21:57:00.679209 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:00.679171    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"]
Oct 30 21:57:00.679413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:00.679375    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" containerID="cri-o://a727d1b8a70d250b05240227fdb6ea7729f15ae9b5034fec1f31eea96d1623af" gracePeriod=30
Oct 30 21:57:01.011039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011004    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"]
Oct 30 21:57:01.011462 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011445    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy"
Oct 30 21:57:01.011518 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011465    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy"
Oct 30 21:57:01.011565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011522    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ff7f4bf0-e541-4893-a000-027b14e42b74" containerName="payload-processing"
Oct 30 21:57:01.011565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011531    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff7f4bf0-e541-4893-a000-027b14e42b74" containerName="payload-processing"
Oct 30 21:57:01.011565 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011563    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a13a507e-48ca-4b4a-90dd-72f30316c75d" containerName="payload-pre-processing"
Oct 30 21:57:01.011677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011570    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13a507e-48ca-4b4a-90dd-72f30316c75d" containerName="payload-pre-processing"
Oct 30 21:57:01.011677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011578    4918 cpu_manager.go:414] "containerMap: removing container" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy"
Oct 30 21:57:01.011677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011583    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy"
Oct 30 21:57:01.011677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011593    4918 cpu_manager.go:414] "containerMap: removing container" podUID="a16b9f2a-0ed2-41ce-b956-a5d5d0081788" containerName="istio-proxy"
Oct 30 21:57:01.011677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011598    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16b9f2a-0ed2-41ce-b956-a5d5d0081788" containerName="istio-proxy"
Oct 30 21:57:01.011677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011610    4918 cpu_manager.go:414] "containerMap: removing container" podUID="ac6c6737-7897-40ba-ba36-6bbcddf7496e" containerName="revoke-keys"
Oct 30 21:57:01.011677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011614    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6c6737-7897-40ba-ba36-6bbcddf7496e" containerName="revoke-keys"
Oct 30 21:57:01.011677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011622    4918 cpu_manager.go:414] "containerMap: removing container" podUID="71dbe041-9629-4ee0-8221-79c215ac06d8" containerName="istio-proxy"
Oct 30 21:57:01.011677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011627    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71dbe041-9629-4ee0-8221-79c215ac06d8" containerName="istio-proxy"
Oct 30 21:57:01.012015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011715    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff7f4bf0-e541-4893-a000-027b14e42b74" containerName="payload-processing"
Oct 30 21:57:01.012015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011725    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13a507e-48ca-4b4a-90dd-72f30316c75d" containerName="payload-pre-processing"
Oct 30 21:57:01.012015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011731    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy"
Oct 30 21:57:01.012015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011738    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16b9f2a-0ed2-41ce-b956-a5d5d0081788" containerName="istio-proxy"
Oct 30 21:57:01.012015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011745    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6c6737-7897-40ba-ba36-6bbcddf7496e" containerName="revoke-keys"
Oct 30 21:57:01.012015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011750    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="71dbe041-9629-4ee0-8221-79c215ac06d8" containerName="istio-proxy"
Oct 30 21:57:01.012015 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.011756    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc03a02c-5042-4f3c-be33-4443682c2f6d" containerName="istio-proxy"
Oct 30 21:57:01.015068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.014973    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:01.023040 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.023023    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:01.023217 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.023203    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:01.024422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.024408    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:01.026412 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.026378    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:57:01.033314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.033289    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"]
Oct 30 21:57:01.075479 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.075444    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg\" (UID: \"9cd8c895-0c42-4163-98bf-f85a9f87d299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:01.075664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.075520    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg\" (UID: \"9cd8c895-0c42-4163-98bf-f85a9f87d299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:01.075664 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.075563    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46nhw\" (UniqueName: \"kubernetes.io/projected/9cd8c895-0c42-4163-98bf-f85a9f87d299-kube-api-access-46nhw\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg\" (UID: \"9cd8c895-0c42-4163-98bf-f85a9f87d299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:01.177157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.177116    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg\" (UID: \"9cd8c895-0c42-4163-98bf-f85a9f87d299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:01.177157 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.177155    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg\" (UID: \"9cd8c895-0c42-4163-98bf-f85a9f87d299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:01.177497 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.177188    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nhw\" (UniqueName: \"kubernetes.io/projected/9cd8c895-0c42-4163-98bf-f85a9f87d299-kube-api-access-46nhw\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg\" (UID: \"9cd8c895-0c42-4163-98bf-f85a9f87d299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:01.177974 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.177949    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg\" (UID: \"9cd8c895-0c42-4163-98bf-f85a9f87d299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:01.179970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.179945    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg\" (UID: \"9cd8c895-0c42-4163-98bf-f85a9f87d299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:01.193451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.193417    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nhw\" (UniqueName: \"kubernetes.io/projected/9cd8c895-0c42-4163-98bf-f85a9f87d299-kube-api-access-46nhw\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg\" (UID: \"9cd8c895-0c42-4163-98bf-f85a9f87d299\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:01.328277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.328191    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:01.610935 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.610908    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"]
Oct 30 21:57:01.612665 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:57:01.612640    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd8c895_0c42_4163_98bf_f85a9f87d299.slice/crio-fca0baa95bab168c6dd03f17e4847b5760f6cb3a597d7f10a20d7d9c7f8b5e49 WatchSource:0}: Error finding container fca0baa95bab168c6dd03f17e4847b5760f6cb3a597d7f10a20d7d9c7f8b5e49: Status 404 returned error can't find the container with id fca0baa95bab168c6dd03f17e4847b5760f6cb3a597d7f10a20d7d9c7f8b5e49
Oct 30 21:57:01.825177 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:01.825145    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg" event={"ID":"9cd8c895-0c42-4163-98bf-f85a9f87d299","Type":"ContainerStarted","Data":"fca0baa95bab168c6dd03f17e4847b5760f6cb3a597d7f10a20d7d9c7f8b5e49"}
Oct 30 21:57:02.453074 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:02.453037    4918 patch_prober.go:29] interesting pod/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:57:02.453270 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:02.453098    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:57:02.834019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:02.833982    4918 generic.go:356] "Generic (PLEG): container finished" podID="9cd8c895-0c42-4163-98bf-f85a9f87d299" containerID="b54b2ff7e6613e3faae39498d1f80af4efe236b9d24fb6375c4a53bc302f03c0" exitCode=0
Oct 30 21:57:02.834482 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:02.834081    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg" event={"ID":"9cd8c895-0c42-4163-98bf-f85a9f87d299","Type":"ContainerDied","Data":"b54b2ff7e6613e3faae39498d1f80af4efe236b9d24fb6375c4a53bc302f03c0"}
Oct 30 21:57:03.990189 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:03.990164    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:04.109892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.109792    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd8c895-0c42-4163-98bf-f85a9f87d299-kube-api-access-46nhw\" (UniqueName: \"kubernetes.io/projected/9cd8c895-0c42-4163-98bf-f85a9f87d299-kube-api-access-46nhw\") pod \"9cd8c895-0c42-4163-98bf-f85a9f87d299\" (UID: \"9cd8c895-0c42-4163-98bf-f85a9f87d299\") "
Oct 30 21:57:04.109892 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.109872    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-xks-ca\") pod \"9cd8c895-0c42-4163-98bf-f85a9f87d299\" (UID: \"9cd8c895-0c42-4163-98bf-f85a9f87d299\") "
Oct 30 21:57:04.110148 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.110001    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-openshift-ca\") pod \"9cd8c895-0c42-4163-98bf-f85a9f87d299\" (UID: \"9cd8c895-0c42-4163-98bf-f85a9f87d299\") "
Oct 30 21:57:04.110437 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.110413    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-openshift-ca" pod "9cd8c895-0c42-4163-98bf-f85a9f87d299" (UID: "9cd8c895-0c42-4163-98bf-f85a9f87d299"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:04.112055 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.112031    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-xks-ca" pod "9cd8c895-0c42-4163-98bf-f85a9f87d299" (UID: "9cd8c895-0c42-4163-98bf-f85a9f87d299"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:04.112175 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.112154    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd8c895-0c42-4163-98bf-f85a9f87d299-kube-api-access-46nhw" pod "9cd8c895-0c42-4163-98bf-f85a9f87d299" (UID: "9cd8c895-0c42-4163-98bf-f85a9f87d299"). InnerVolumeSpecName "kube-api-access-46nhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:04.211972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.211938    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.211972 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.211973    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46nhw\" (UniqueName: \"kubernetes.io/projected/9cd8c895-0c42-4163-98bf-f85a9f87d299-kube-api-access-46nhw\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.212239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.211988    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd8c895-0c42-4163-98bf-f85a9f87d299-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.848834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.848805    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"
Oct 30 21:57:04.848834 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.848820    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg" event={"ID":"9cd8c895-0c42-4163-98bf-f85a9f87d299","Type":"ContainerDied","Data":"fca0baa95bab168c6dd03f17e4847b5760f6cb3a597d7f10a20d7d9c7f8b5e49"}
Oct 30 21:57:04.849059 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.848864    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fca0baa95bab168c6dd03f17e4847b5760f6cb3a597d7f10a20d7d9c7f8b5e49"
Oct 30 21:57:04.855846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:04.855810    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb" podUID="836991d9-b095-44e2-83af-b8a260eb2413" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:57:05.802184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.802160    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:57:05.859473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.859442    4918 generic.go:356] "Generic (PLEG): container finished" podID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" containerID="4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37" exitCode=0
Oct 30 21:57:05.859638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.859525    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" event={"ID":"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870","Type":"ContainerDied","Data":"4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37"}
Oct 30 21:57:05.859638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.859545    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"
Oct 30 21:57:05.859638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.859590    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp" event={"ID":"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870","Type":"ContainerDied","Data":"14ad47c6f938611f3fe5051e370d83576159093aeea93187e97b866b94d37541"}
Oct 30 21:57:05.859638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.859621    4918 scope.go:122] "RemoveContainer" containerID="4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37"
Oct 30 21:57:05.861707 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.861681    4918 generic.go:356] "Generic (PLEG): container finished" podID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerID="a727d1b8a70d250b05240227fdb6ea7729f15ae9b5034fec1f31eea96d1623af" exitCode=0
Oct 30 21:57:05.861843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.861774    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" event={"ID":"dfd9f7af-1f13-4301-b393-fe62a8029ea2","Type":"ContainerDied","Data":"a727d1b8a70d250b05240227fdb6ea7729f15ae9b5034fec1f31eea96d1623af"}
Oct 30 21:57:05.872167 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.872149    4918 scope.go:122] "RemoveContainer" containerID="4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37"
Oct 30 21:57:05.873483 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:57:05.873452    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37\": container with ID starting with 4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37 not found: ID does not exist" containerID="4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37"
Oct 30 21:57:05.873606 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.873495    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37"} err="failed to get container status \"4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37\": rpc error: code = NotFound desc = could not find container \"4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37\": container with ID starting with 4d0d724cfa78a48296521656b1c216e098c69995cc91ec105f00f7426a772e37 not found: ID does not exist"
Oct 30 21:57:05.931615 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.931588    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-envoy\") pod \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") "
Oct 30 21:57:05.931750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.931624    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-kube-api-access-qldxl\" (UniqueName: \"kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-kube-api-access-qldxl\") pod \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") "
Oct 30 21:57:05.931750 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.931723    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-podinfo\") pod \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") "
Oct 30 21:57:05.931857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.931751    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-token\" (UniqueName: \"kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-token\") pod \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") "
Oct 30 21:57:05.931857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.931778    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-credential-socket\") pod \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") "
Oct 30 21:57:05.931857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.931815    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-data\") pod \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") "
Oct 30 21:57:05.931857 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.931846    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-certs\") pod \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") "
Oct 30 21:57:05.932101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.931901    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istiod-ca-cert\") pod \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") "
Oct 30 21:57:05.932101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.932001    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-socket\") pod \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\" (UID: \"5c890ea5-9ab7-4c1f-86bc-a13c55e9e870\") "
Oct 30 21:57:05.932216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.932124    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-credential-socket" pod "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" (UID: "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.932216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.932160    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-data" pod "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" (UID: "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.932216 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.932204    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-certs" pod "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" (UID: "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.932596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.932575    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.932596 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.932587    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-socket" pod "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" (UID: "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.932699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.932604    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.932699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.932619    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.932788 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.932701    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istiod-ca-cert" pod "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" (UID: "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:05.934100 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.934079    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-envoy" pod "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" (UID: "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.934233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.934134    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-token" pod "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" (UID: "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:05.934575 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.934559    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-kube-api-access-qldxl" pod "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" (UID: "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870"). InnerVolumeSpecName "kube-api-access-qldxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:05.934930 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.934907    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-podinfo" pod "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" (UID: "5c890ea5-9ab7-4c1f-86bc-a13c55e9e870"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:05.947237 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:05.947215    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:57:06.033765 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.033737    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istiod-ca-cert\") pod \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") "
Oct 30 21:57:06.033994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.033775    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-token\" (UniqueName: \"kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-token\") pod \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") "
Oct 30 21:57:06.033994 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.033815    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-envoy\") pod \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") "
Oct 30 21:57:06.034102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034001    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-socket\") pod \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") "
Oct 30 21:57:06.034102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034056    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-credential-socket\") pod \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") "
Oct 30 21:57:06.034102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034095    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-certs\") pod \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") "
Oct 30 21:57:06.034268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034212    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istiod-ca-cert" pod "dfd9f7af-1f13-4301-b393-fe62a8029ea2" (UID: "dfd9f7af-1f13-4301-b393-fe62a8029ea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:06.034268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034228    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-podinfo\") pod \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") "
Oct 30 21:57:06.034268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034241    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-socket" pod "dfd9f7af-1f13-4301-b393-fe62a8029ea2" (UID: "dfd9f7af-1f13-4301-b393-fe62a8029ea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:06.034467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034285    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-kube-api-access-rdpt8\" (UniqueName: \"kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-kube-api-access-rdpt8\") pod \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") "
Oct 30 21:57:06.034467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034326    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-credential-socket" pod "dfd9f7af-1f13-4301-b393-fe62a8029ea2" (UID: "dfd9f7af-1f13-4301-b393-fe62a8029ea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:06.034467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034335    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-data\") pod \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\" (UID: \"dfd9f7af-1f13-4301-b393-fe62a8029ea2\") "
Oct 30 21:57:06.034467 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034365    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-certs" pod "dfd9f7af-1f13-4301-b393-fe62a8029ea2" (UID: "dfd9f7af-1f13-4301-b393-fe62a8029ea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:06.034868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034585    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-data" pod "dfd9f7af-1f13-4301-b393-fe62a8029ea2" (UID: "dfd9f7af-1f13-4301-b393-fe62a8029ea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:06.034868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034838    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.034868 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034860    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.035054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034876    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.035054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034890    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.035054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034906    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.035054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034921    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qldxl\" (UniqueName: \"kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-kube-api-access-qldxl\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.035054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034936    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.035054 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.034951    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.035413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.035058    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.035413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.035072    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.035413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.035085    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.036691 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.036665    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-token" pod "dfd9f7af-1f13-4301-b393-fe62a8029ea2" (UID: "dfd9f7af-1f13-4301-b393-fe62a8029ea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:06.036895 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.036870    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-envoy" pod "dfd9f7af-1f13-4301-b393-fe62a8029ea2" (UID: "dfd9f7af-1f13-4301-b393-fe62a8029ea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:06.037020 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.036923    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-podinfo" pod "dfd9f7af-1f13-4301-b393-fe62a8029ea2" (UID: "dfd9f7af-1f13-4301-b393-fe62a8029ea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:06.037088 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.037058    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-kube-api-access-rdpt8" pod "dfd9f7af-1f13-4301-b393-fe62a8029ea2" (UID: "dfd9f7af-1f13-4301-b393-fe62a8029ea2"). InnerVolumeSpecName "kube-api-access-rdpt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:06.136642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.136557    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.136642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.136586    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.136642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.136596    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfd9f7af-1f13-4301-b393-fe62a8029ea2-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.136642 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.136607    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdpt8\" (UniqueName: \"kubernetes.io/projected/dfd9f7af-1f13-4301-b393-fe62a8029ea2-kube-api-access-rdpt8\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.187416 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.187326    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"]
Oct 30 21:57:06.190381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.190351    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0ea1ea-openshift-default-78d5f77b9-gjmnp"]
Oct 30 21:57:06.874803 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.874705    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg" event={"ID":"dfd9f7af-1f13-4301-b393-fe62a8029ea2","Type":"ContainerDied","Data":"eeafc86573e42ecf96b2406c374a8259d1112cf7175584ac119932d8c1a04dcc"}
Oct 30 21:57:06.875298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.874898    4918 scope.go:122] "RemoveContainer" containerID="a727d1b8a70d250b05240227fdb6ea7729f15ae9b5034fec1f31eea96d1623af"
Oct 30 21:57:06.875298 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.874652    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"
Oct 30 21:57:06.902019 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.901994    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"]
Oct 30 21:57:06.906897 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:06.906874    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a33b12-openshift-default-749cf497bc-4tvvg"]
Oct 30 21:57:07.104774 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:07.104736    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" path="/var/lib/kubelet/pods/5c890ea5-9ab7-4c1f-86bc-a13c55e9e870/volumes"
Oct 30 21:57:07.105214 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:07.105201    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" path="/var/lib/kubelet/pods/dfd9f7af-1f13-4301-b393-fe62a8029ea2/volumes"
Oct 30 21:57:08.510494 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.510467    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"
Oct 30 21:57:08.663169 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.663078    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836991d9-b095-44e2-83af-b8a260eb2413-kube-api-access-swppg\" (UniqueName: \"kubernetes.io/projected/836991d9-b095-44e2-83af-b8a260eb2413-kube-api-access-swppg\") pod \"836991d9-b095-44e2-83af-b8a260eb2413\" (UID: \"836991d9-b095-44e2-83af-b8a260eb2413\") "
Oct 30 21:57:08.663351 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.663207    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836991d9-b095-44e2-83af-b8a260eb2413-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836991d9-b095-44e2-83af-b8a260eb2413-plugins-config-volume\") pod \"836991d9-b095-44e2-83af-b8a260eb2413\" (UID: \"836991d9-b095-44e2-83af-b8a260eb2413\") "
Oct 30 21:57:08.663603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.663583    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836991d9-b095-44e2-83af-b8a260eb2413-plugins-config-volume" pod "836991d9-b095-44e2-83af-b8a260eb2413" (UID: "836991d9-b095-44e2-83af-b8a260eb2413"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:08.665473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.665441    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836991d9-b095-44e2-83af-b8a260eb2413-kube-api-access-swppg" pod "836991d9-b095-44e2-83af-b8a260eb2413" (UID: "836991d9-b095-44e2-83af-b8a260eb2413"). InnerVolumeSpecName "kube-api-access-swppg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:08.764262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.764223    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swppg\" (UniqueName: \"kubernetes.io/projected/836991d9-b095-44e2-83af-b8a260eb2413-kube-api-access-swppg\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:08.764262 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.764260    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836991d9-b095-44e2-83af-b8a260eb2413-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:08.891704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.891664    4918 generic.go:356] "Generic (PLEG): container finished" podID="836991d9-b095-44e2-83af-b8a260eb2413" containerID="ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650" exitCode=0
Oct 30 21:57:08.891861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.891716    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"
Oct 30 21:57:08.891861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.891742    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb" event={"ID":"836991d9-b095-44e2-83af-b8a260eb2413","Type":"ContainerDied","Data":"ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650"}
Oct 30 21:57:08.891861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.891800    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb" event={"ID":"836991d9-b095-44e2-83af-b8a260eb2413","Type":"ContainerDied","Data":"9781e04445ae6fdc1bf5ab18173d6fa52578166e2c7c74907dfdd1431ca19bb7"}
Oct 30 21:57:08.891861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.891823    4918 scope.go:122] "RemoveContainer" containerID="ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650"
Oct 30 21:57:08.902373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.902355    4918 scope.go:122] "RemoveContainer" containerID="ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650"
Oct 30 21:57:08.902673 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:57:08.902654    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650\": container with ID starting with ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650 not found: ID does not exist" containerID="ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650"
Oct 30 21:57:08.902737 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.902682    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650"} err="failed to get container status \"ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650\": rpc error: code = NotFound desc = could not find container \"ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650\": container with ID starting with ff3ed003bb29fadd8b6336a77f165dfa1ac4def9ca8bf9011e431fdd176b3650 not found: ID does not exist"
Oct 30 21:57:08.916650 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.916591    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"]
Oct 30 21:57:08.922799 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:08.922777    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0ea1ea-64d4fcc799-xl8hb"]
Oct 30 21:57:09.031780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:09.031709    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"]
Oct 30 21:57:09.034492 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:09.034466    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98394fc5-jczws"]
Oct 30 21:57:09.076521 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:09.076491    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010ec348-3f3c-46f5-ba7f-55ebbf70029b" path="/var/lib/kubelet/pods/010ec348-3f3c-46f5-ba7f-55ebbf70029b/volumes"
Oct 30 21:57:09.076976 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:09.076956    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836991d9-b095-44e2-83af-b8a260eb2413" path="/var/lib/kubelet/pods/836991d9-b095-44e2-83af-b8a260eb2413/volumes"
Oct 30 21:57:11.127127 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:11.127094    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"]
Oct 30 21:57:14.527539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:14.527502    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"]
Oct 30 21:57:14.532365 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:14.532336    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-06bb0f-k59fg"]
Oct 30 21:57:15.076238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:15.076204    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd8c895-0c42-4163-98bf-f85a9f87d299" path="/var/lib/kubelet/pods/9cd8c895-0c42-4163-98bf-f85a9f87d299/volumes"
Oct 30 21:57:16.127716 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:16.127675    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr" podUID="87183b40-b615-4b61-b9bd-a26fb3da3d7a" containerName="payload-pre-processing" containerID="cri-o://4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1" gracePeriod=25
Oct 30 21:57:20.021604 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:20.021228    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"]
Oct 30 21:57:20.024170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:20.024147    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2bc375c-2xr8m"]
Oct 30 21:57:21.076580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:21.076546    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc44b1a9-3531-4107-8960-d4c7fbc51757" path="/var/lib/kubelet/pods/fc44b1a9-3531-4107-8960-d4c7fbc51757/volumes"
Oct 30 21:57:22.397208 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.397175    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"]
Oct 30 21:57:22.397675 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.397374    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" podUID="f0f22573-6b3d-467b-978c-d37e79634546" containerName="istio-proxy" containerID="cri-o://e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557" gracePeriod=30
Oct 30 21:57:22.878553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.878520    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"]
Oct 30 21:57:22.878965 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.878951    4918 cpu_manager.go:414] "containerMap: removing container" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy"
Oct 30 21:57:22.879039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.878967    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy"
Oct 30 21:57:22.879039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.878980    4918 cpu_manager.go:414] "containerMap: removing container" podUID="9cd8c895-0c42-4163-98bf-f85a9f87d299" containerName="revoke-keys"
Oct 30 21:57:22.879039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.878985    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd8c895-0c42-4163-98bf-f85a9f87d299" containerName="revoke-keys"
Oct 30 21:57:22.879039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.879003    4918 cpu_manager.go:414] "containerMap: removing container" podUID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" containerName="istio-proxy"
Oct 30 21:57:22.879039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.879010    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" containerName="istio-proxy"
Oct 30 21:57:22.879039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.879034    4918 cpu_manager.go:414] "containerMap: removing container" podUID="836991d9-b095-44e2-83af-b8a260eb2413" containerName="payload-processing"
Oct 30 21:57:22.879039 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.879039    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836991d9-b095-44e2-83af-b8a260eb2413" containerName="payload-processing"
Oct 30 21:57:22.879310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.879098    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="836991d9-b095-44e2-83af-b8a260eb2413" containerName="payload-processing"
Oct 30 21:57:22.879310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.879109    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd9f7af-1f13-4301-b393-fe62a8029ea2" containerName="istio-proxy"
Oct 30 21:57:22.879310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.879115    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd8c895-0c42-4163-98bf-f85a9f87d299" containerName="revoke-keys"
Oct 30 21:57:22.879310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.879123    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c890ea5-9ab7-4c1f-86bc-a13c55e9e870" containerName="istio-proxy"
Oct 30 21:57:22.883678 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.883662    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:22.889372 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.888825    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:22.895160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.889812    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:22.895160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.889879    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:22.895160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.890246    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:57:22.895160 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.893627    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"]
Oct 30 21:57:22.903883 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.903856    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg\" (UID: \"bde7ba46-9662-4d97-938d-1c294fb5af79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:22.904032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.903888    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg\" (UID: \"bde7ba46-9662-4d97-938d-1c294fb5af79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:22.904032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:22.903920    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqzjz\" (UniqueName: \"kubernetes.io/projected/bde7ba46-9662-4d97-938d-1c294fb5af79-kube-api-access-wqzjz\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg\" (UID: \"bde7ba46-9662-4d97-938d-1c294fb5af79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:23.006032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:23.005983    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg\" (UID: \"bde7ba46-9662-4d97-938d-1c294fb5af79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:23.006859 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:23.006832    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg\" (UID: \"bde7ba46-9662-4d97-938d-1c294fb5af79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:23.007115 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:23.006879    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqzjz\" (UniqueName: \"kubernetes.io/projected/bde7ba46-9662-4d97-938d-1c294fb5af79-kube-api-access-wqzjz\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg\" (UID: \"bde7ba46-9662-4d97-938d-1c294fb5af79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:23.010034 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:23.010005    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg\" (UID: \"bde7ba46-9662-4d97-938d-1c294fb5af79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:23.012083 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:23.012054    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg\" (UID: \"bde7ba46-9662-4d97-938d-1c294fb5af79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:23.019711 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:23.019686    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqzjz\" (UniqueName: \"kubernetes.io/projected/bde7ba46-9662-4d97-938d-1c294fb5af79-kube-api-access-wqzjz\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg\" (UID: \"bde7ba46-9662-4d97-938d-1c294fb5af79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:23.201464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:23.201361    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:23.484961 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:23.484930    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"]
Oct 30 21:57:23.485849 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:57:23.485821    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde7ba46_9662_4d97_938d_1c294fb5af79.slice/crio-fad57389b1249b4a6ad43b6b2133491eb07edf29cd723f564aded236617306d0 WatchSource:0}: Error finding container fad57389b1249b4a6ad43b6b2133491eb07edf29cd723f564aded236617306d0: Status 404 returned error can't find the container with id fad57389b1249b4a6ad43b6b2133491eb07edf29cd723f564aded236617306d0
Oct 30 21:57:23.973622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:23.973590    4918 patch_prober.go:29] interesting pod/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:57:23.973796 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:23.973652    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" podUID="f0f22573-6b3d-467b-978c-d37e79634546" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:57:24.008101 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:24.008066    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg" event={"ID":"bde7ba46-9662-4d97-938d-1c294fb5af79","Type":"ContainerStarted","Data":"5b666beff04e7eb446de90f2b640c58b6f9c2fd9bc3e6ef5b6726d7838746315"}
Oct 30 21:57:24.008251 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:24.008122    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg" event={"ID":"bde7ba46-9662-4d97-938d-1c294fb5af79","Type":"ContainerStarted","Data":"fad57389b1249b4a6ad43b6b2133491eb07edf29cd723f564aded236617306d0"}
Oct 30 21:57:24.029381 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:24.029333    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg" podStartSLOduration=2.029319781 podStartE2EDuration="2.029319781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:24.026968086 +0000 UTC m=+3357.073527231" watchObservedRunningTime="2026-10-30 21:57:24.029319781 +0000 UTC m=+3357.075878949"
Oct 30 21:57:25.016382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:25.016340    4918 generic.go:356] "Generic (PLEG): container finished" podID="bde7ba46-9662-4d97-938d-1c294fb5af79" containerID="5b666beff04e7eb446de90f2b640c58b6f9c2fd9bc3e6ef5b6726d7838746315" exitCode=0
Oct 30 21:57:25.016867 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:25.016431    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg" event={"ID":"bde7ba46-9662-4d97-938d-1c294fb5af79","Type":"ContainerDied","Data":"5b666beff04e7eb446de90f2b640c58b6f9c2fd9bc3e6ef5b6726d7838746315"}
Oct 30 21:57:26.038681 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:26.038644    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr" podUID="87183b40-b615-4b61-b9bd-a26fb3da3d7a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:57:26.164188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:26.164163    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:26.246121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:26.246090    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde7ba46-9662-4d97-938d-1c294fb5af79-kube-api-access-wqzjz\" (UniqueName: \"kubernetes.io/projected/bde7ba46-9662-4d97-938d-1c294fb5af79-kube-api-access-wqzjz\") pod \"bde7ba46-9662-4d97-938d-1c294fb5af79\" (UID: \"bde7ba46-9662-4d97-938d-1c294fb5af79\") "
Oct 30 21:57:26.246296 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:26.246193    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-xks-ca\") pod \"bde7ba46-9662-4d97-938d-1c294fb5af79\" (UID: \"bde7ba46-9662-4d97-938d-1c294fb5af79\") "
Oct 30 21:57:26.246296 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:26.246283    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-openshift-ca\") pod \"bde7ba46-9662-4d97-938d-1c294fb5af79\" (UID: \"bde7ba46-9662-4d97-938d-1c294fb5af79\") "
Oct 30 21:57:26.246679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:26.246649    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-openshift-ca" pod "bde7ba46-9662-4d97-938d-1c294fb5af79" (UID: "bde7ba46-9662-4d97-938d-1c294fb5af79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:26.248451 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:26.248427    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde7ba46-9662-4d97-938d-1c294fb5af79-kube-api-access-wqzjz" pod "bde7ba46-9662-4d97-938d-1c294fb5af79" (UID: "bde7ba46-9662-4d97-938d-1c294fb5af79"). InnerVolumeSpecName "kube-api-access-wqzjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:26.248704 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:26.248677    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-xks-ca" pod "bde7ba46-9662-4d97-938d-1c294fb5af79" (UID: "bde7ba46-9662-4d97-938d-1c294fb5af79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:26.348009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:26.347929    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:26.348009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:26.347958    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqzjz\" (UniqueName: \"kubernetes.io/projected/bde7ba46-9662-4d97-938d-1c294fb5af79-kube-api-access-wqzjz\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:26.348009 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:26.347969    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bde7ba46-9662-4d97-938d-1c294fb5af79-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.032188 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.032155    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg" event={"ID":"bde7ba46-9662-4d97-938d-1c294fb5af79","Type":"ContainerDied","Data":"fad57389b1249b4a6ad43b6b2133491eb07edf29cd723f564aded236617306d0"}
Oct 30 21:57:27.032359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.032199    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"
Oct 30 21:57:27.032359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.032208    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad57389b1249b4a6ad43b6b2133491eb07edf29cd723f564aded236617306d0"
Oct 30 21:57:27.660119 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.660056    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:57:27.759738 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.759705    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-certs\") pod \"f0f22573-6b3d-467b-978c-d37e79634546\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") "
Oct 30 21:57:27.759940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.759752    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-istio-token\" (UniqueName: \"kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-istio-token\") pod \"f0f22573-6b3d-467b-978c-d37e79634546\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") "
Oct 30 21:57:27.759940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.759800    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-socket\") pod \"f0f22573-6b3d-467b-978c-d37e79634546\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") "
Oct 30 21:57:27.759940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.759854    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-envoy\") pod \"f0f22573-6b3d-467b-978c-d37e79634546\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") "
Oct 30 21:57:27.760124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.759977    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-kube-api-access-dtl75\" (UniqueName: \"kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-kube-api-access-dtl75\") pod \"f0f22573-6b3d-467b-978c-d37e79634546\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") "
Oct 30 21:57:27.760124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.760032    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-certs" pod "f0f22573-6b3d-467b-978c-d37e79634546" (UID: "f0f22573-6b3d-467b-978c-d37e79634546"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:27.760124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.760069    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0f22573-6b3d-467b-978c-d37e79634546-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f22573-6b3d-467b-978c-d37e79634546-istio-podinfo\") pod \"f0f22573-6b3d-467b-978c-d37e79634546\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") "
Oct 30 21:57:27.760124 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.760116    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-data\") pod \"f0f22573-6b3d-467b-978c-d37e79634546\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") "
Oct 30 21:57:27.760350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.760137    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-socket" pod "f0f22573-6b3d-467b-978c-d37e79634546" (UID: "f0f22573-6b3d-467b-978c-d37e79634546"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:27.760350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.760160    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f22573-6b3d-467b-978c-d37e79634546-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f22573-6b3d-467b-978c-d37e79634546-istiod-ca-cert\") pod \"f0f22573-6b3d-467b-978c-d37e79634546\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") "
Oct 30 21:57:27.760350 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.760209    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-credential-socket\") pod \"f0f22573-6b3d-467b-978c-d37e79634546\" (UID: \"f0f22573-6b3d-467b-978c-d37e79634546\") "
Oct 30 21:57:27.760569 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.760366    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-data" pod "f0f22573-6b3d-467b-978c-d37e79634546" (UID: "f0f22573-6b3d-467b-978c-d37e79634546"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:27.760662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.760624    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-credential-socket" pod "f0f22573-6b3d-467b-978c-d37e79634546" (UID: "f0f22573-6b3d-467b-978c-d37e79634546"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:27.760736 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.760663    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f22573-6b3d-467b-978c-d37e79634546-istiod-ca-cert" pod "f0f22573-6b3d-467b-978c-d37e79634546" (UID: "f0f22573-6b3d-467b-978c-d37e79634546"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:27.760846 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.760831    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.760915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.760866    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.760915 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.760883    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.762359 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.762330    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0f22573-6b3d-467b-978c-d37e79634546-istio-podinfo" pod "f0f22573-6b3d-467b-978c-d37e79634546" (UID: "f0f22573-6b3d-467b-978c-d37e79634546"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:27.762501 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.762402    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-kube-api-access-dtl75" pod "f0f22573-6b3d-467b-978c-d37e79634546" (UID: "f0f22573-6b3d-467b-978c-d37e79634546"). InnerVolumeSpecName "kube-api-access-dtl75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:27.762661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.762636    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-envoy" pod "f0f22573-6b3d-467b-978c-d37e79634546" (UID: "f0f22573-6b3d-467b-978c-d37e79634546"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:27.762661 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.762654    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-istio-token" pod "f0f22573-6b3d-467b-978c-d37e79634546" (UID: "f0f22573-6b3d-467b-978c-d37e79634546"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:27.861655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.861622    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.861655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.861649    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.861655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.861661    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f22573-6b3d-467b-978c-d37e79634546-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.861922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.861671    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtl75\" (UniqueName: \"kubernetes.io/projected/f0f22573-6b3d-467b-978c-d37e79634546-kube-api-access-dtl75\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.861922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.861682    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f22573-6b3d-467b-978c-d37e79634546-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.861922 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:27.861690    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f22573-6b3d-467b-978c-d37e79634546-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:28.039903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:28.039868    4918 generic.go:356] "Generic (PLEG): container finished" podID="f0f22573-6b3d-467b-978c-d37e79634546" containerID="e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557" exitCode=0
Oct 30 21:57:28.040063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:28.039917    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"
Oct 30 21:57:28.040063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:28.039953    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" event={"ID":"f0f22573-6b3d-467b-978c-d37e79634546","Type":"ContainerDied","Data":"e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557"}
Oct 30 21:57:28.040063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:28.040009    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k" event={"ID":"f0f22573-6b3d-467b-978c-d37e79634546","Type":"ContainerDied","Data":"e02c5e5e72a706054b7d690a290b172dcec9df4c4074d48347873ce82cdaa941"}
Oct 30 21:57:28.040063 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:28.040040    4918 scope.go:122] "RemoveContainer" containerID="e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557"
Oct 30 21:57:28.050953 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:28.050937    4918 scope.go:122] "RemoveContainer" containerID="e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557"
Oct 30 21:57:28.051257 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:57:28.051234    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557\": container with ID starting with e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557 not found: ID does not exist" containerID="e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557"
Oct 30 21:57:28.051330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:28.051267    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557"} err="failed to get container status \"e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557\": rpc error: code = NotFound desc = could not find container \"e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557\": container with ID starting with e36377a7380e153e572b6779696c93e1393cb142fb6295ce7d2d3b44dacdd557 not found: ID does not exist"
Oct 30 21:57:28.070734 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:28.070707    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"]
Oct 30 21:57:28.083103 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:28.083083    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-06bb0f-openshift-default-749ff4f4b7-24t2k"]
Oct 30 21:57:29.075011 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:29.074977    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f22573-6b3d-467b-978c-d37e79634546" path="/var/lib/kubelet/pods/f0f22573-6b3d-467b-978c-d37e79634546/volumes"
Oct 30 21:57:29.157469 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:57:29.157431    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:57:35.368245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:35.368212    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"]
Oct 30 21:57:35.376940 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:35.376912    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6ed261-wh2zg"]
Oct 30 21:57:36.038934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:36.038895    4918 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr" podUID="87183b40-b615-4b61-b9bd-a26fb3da3d7a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:57:36.389516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:36.389488    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"
Oct 30 21:57:36.447241 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:36.447215    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87183b40-b615-4b61-b9bd-a26fb3da3d7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87183b40-b615-4b61-b9bd-a26fb3da3d7a-plugins-config-volume\") pod \"87183b40-b615-4b61-b9bd-a26fb3da3d7a\" (UID: \"87183b40-b615-4b61-b9bd-a26fb3da3d7a\") "
Oct 30 21:57:36.447421 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:36.447335    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87183b40-b615-4b61-b9bd-a26fb3da3d7a-kube-api-access-xrp2d\" (UniqueName: \"kubernetes.io/projected/87183b40-b615-4b61-b9bd-a26fb3da3d7a-kube-api-access-xrp2d\") pod \"87183b40-b615-4b61-b9bd-a26fb3da3d7a\" (UID: \"87183b40-b615-4b61-b9bd-a26fb3da3d7a\") "
Oct 30 21:57:36.447592 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:36.447569    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87183b40-b615-4b61-b9bd-a26fb3da3d7a-plugins-config-volume" pod "87183b40-b615-4b61-b9bd-a26fb3da3d7a" (UID: "87183b40-b615-4b61-b9bd-a26fb3da3d7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:36.449744 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:36.449716    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87183b40-b615-4b61-b9bd-a26fb3da3d7a-kube-api-access-xrp2d" pod "87183b40-b615-4b61-b9bd-a26fb3da3d7a" (UID: "87183b40-b615-4b61-b9bd-a26fb3da3d7a"). InnerVolumeSpecName "kube-api-access-xrp2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:36.549170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:36.549091    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrp2d\" (UniqueName: \"kubernetes.io/projected/87183b40-b615-4b61-b9bd-a26fb3da3d7a-kube-api-access-xrp2d\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:36.549170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:36.549118    4918 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87183b40-b615-4b61-b9bd-a26fb3da3d7a-plugins-config-volume\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:37.075590 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:37.075516    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde7ba46-9662-4d97-938d-1c294fb5af79" path="/var/lib/kubelet/pods/bde7ba46-9662-4d97-938d-1c294fb5af79/volumes"
Oct 30 21:57:37.107342 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:37.107304    4918 generic.go:356] "Generic (PLEG): container finished" podID="87183b40-b615-4b61-b9bd-a26fb3da3d7a" containerID="4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1" exitCode=0
Oct 30 21:57:37.107543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:37.107365    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"
Oct 30 21:57:37.107543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:37.107373    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr" event={"ID":"87183b40-b615-4b61-b9bd-a26fb3da3d7a","Type":"ContainerDied","Data":"4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1"}
Oct 30 21:57:37.107543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:37.107469    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr" event={"ID":"87183b40-b615-4b61-b9bd-a26fb3da3d7a","Type":"ContainerDied","Data":"3d40357d4f7c7e4a482da5e186ba6490baa8e949a8714facc76d6bd78980b6fd"}
Oct 30 21:57:37.107543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:37.107504    4918 scope.go:122] "RemoveContainer" containerID="4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1"
Oct 30 21:57:37.117666 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:37.117633    4918 scope.go:122] "RemoveContainer" containerID="4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1"
Oct 30 21:57:37.117958 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:57:37.117940    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1\": container with ID starting with 4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1 not found: ID does not exist" containerID="4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1"
Oct 30 21:57:37.118033 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:37.117966    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1"} err="failed to get container status \"4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1\": rpc error: code = NotFound desc = could not find container \"4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1\": container with ID starting with 4e5d974de1ad0f3115b8808bcb6a5d359f5430feb0c37d678da64883f49a03a1 not found: ID does not exist"
Oct 30 21:57:37.131361 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:37.131337    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"]
Oct 30 21:57:37.140422 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:37.140378    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-06bb0f-8547bc496c-vdhgr"]
Oct 30 21:57:39.077147 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:39.077116    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87183b40-b615-4b61-b9bd-a26fb3da3d7a" path="/var/lib/kubelet/pods/87183b40-b615-4b61-b9bd-a26fb3da3d7a/volumes"
Oct 30 21:57:43.672903 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:43.672869    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"]
Oct 30 21:57:43.673353 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:43.673122    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy" containerID="cri-o://3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c" gracePeriod=30
Oct 30 21:57:44.403141 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.403111    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"]
Oct 30 21:57:44.403568 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.403553    4918 cpu_manager.go:414] "containerMap: removing container" podUID="bde7ba46-9662-4d97-938d-1c294fb5af79" containerName="revoke-keys"
Oct 30 21:57:44.403629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.403571    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde7ba46-9662-4d97-938d-1c294fb5af79" containerName="revoke-keys"
Oct 30 21:57:44.403629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.403589    4918 cpu_manager.go:414] "containerMap: removing container" podUID="f0f22573-6b3d-467b-978c-d37e79634546" containerName="istio-proxy"
Oct 30 21:57:44.403629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.403594    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f22573-6b3d-467b-978c-d37e79634546" containerName="istio-proxy"
Oct 30 21:57:44.403629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.403609    4918 cpu_manager.go:414] "containerMap: removing container" podUID="87183b40-b615-4b61-b9bd-a26fb3da3d7a" containerName="payload-pre-processing"
Oct 30 21:57:44.403629 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.403615    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87183b40-b615-4b61-b9bd-a26fb3da3d7a" containerName="payload-pre-processing"
Oct 30 21:57:44.403828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.403676    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f22573-6b3d-467b-978c-d37e79634546" containerName="istio-proxy"
Oct 30 21:57:44.403828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.403686    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="87183b40-b615-4b61-b9bd-a26fb3da3d7a" containerName="payload-pre-processing"
Oct 30 21:57:44.403828 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.403695    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde7ba46-9662-4d97-938d-1c294fb5af79" containerName="revoke-keys"
Oct 30 21:57:44.408283 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.408267    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:44.418090 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.418072    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:44.418238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.418136    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:44.418238 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.418232    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:57:44.418468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.418314    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:44.419964 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.419945    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"]
Oct 30 21:57:44.533052 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.533015    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl\" (UID: \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:44.533239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.533073    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl\" (UID: \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:44.533239 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.533151    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvdkd\" (UniqueName: \"kubernetes.io/projected/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-kube-api-access-gvdkd\") pod \"maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl\" (UID: \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:44.634332 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.634301    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvdkd\" (UniqueName: \"kubernetes.io/projected/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-kube-api-access-gvdkd\") pod \"maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl\" (UID: \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:44.634539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.634432    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl\" (UID: \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:44.634539 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.634468    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl\" (UID: \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:44.635061 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.635040    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl\" (UID: \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:44.636938 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.636913    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl\" (UID: \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:44.645950 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.645927    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvdkd\" (UniqueName: \"kubernetes.io/projected/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-kube-api-access-gvdkd\") pod \"maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl\" (UID: \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:44.723048 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:44.723019    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:45.020065 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:45.020037    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"]
Oct 30 21:57:45.021981 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:57:45.021955    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b795ed_ce3b_4c15_9b5b_33ceb4f83680.slice/crio-0248f801e136f40acdf92bcd8c5085a4491685287bcf41f38080c8158d63d667 WatchSource:0}: Error finding container 0248f801e136f40acdf92bcd8c5085a4491685287bcf41f38080c8158d63d667: Status 404 returned error can't find the container with id 0248f801e136f40acdf92bcd8c5085a4491685287bcf41f38080c8158d63d667
Oct 30 21:57:45.160553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:45.160518    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl" event={"ID":"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680","Type":"ContainerStarted","Data":"0248f801e136f40acdf92bcd8c5085a4491685287bcf41f38080c8158d63d667"}
Oct 30 21:57:46.169433 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:46.169322    4918 generic.go:356] "Generic (PLEG): container finished" podID="d6b795ed-ce3b-4c15-9b5b-33ceb4f83680" containerID="f5e31eb50d795ceee6295a9b1bb7725aa03ea2986b68ce44de023a1db8b9b811" exitCode=0
Oct 30 21:57:46.169847 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:46.169425    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl" event={"ID":"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680","Type":"ContainerDied","Data":"f5e31eb50d795ceee6295a9b1bb7725aa03ea2986b68ce44de023a1db8b9b811"}
Oct 30 21:57:47.310941 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:47.310918    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:47.464222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:47.464136    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-kube-api-access-gvdkd\" (UniqueName: \"kubernetes.io/projected/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-kube-api-access-gvdkd\") pod \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\" (UID: \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\") "
Oct 30 21:57:47.464222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:47.464177    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-openshift-ca\") pod \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\" (UID: \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\") "
Oct 30 21:57:47.464222 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:47.464221    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-xks-ca\") pod \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\" (UID: \"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680\") "
Oct 30 21:57:47.464677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:47.464650    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-openshift-ca" pod "d6b795ed-ce3b-4c15-9b5b-33ceb4f83680" (UID: "d6b795ed-ce3b-4c15-9b5b-33ceb4f83680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:47.466485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:47.466462    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-xks-ca" pod "d6b795ed-ce3b-4c15-9b5b-33ceb4f83680" (UID: "d6b795ed-ce3b-4c15-9b5b-33ceb4f83680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:47.466589 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:47.466569    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-kube-api-access-gvdkd" pod "d6b795ed-ce3b-4c15-9b5b-33ceb4f83680" (UID: "d6b795ed-ce3b-4c15-9b5b-33ceb4f83680"). InnerVolumeSpecName "kube-api-access-gvdkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:47.565263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:47.565231    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvdkd\" (UniqueName: \"kubernetes.io/projected/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-kube-api-access-gvdkd\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:47.565263 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:47.565261    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:47.565485 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:47.565271    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:48.184314 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:48.184283    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"
Oct 30 21:57:48.184516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:48.184309    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl" event={"ID":"d6b795ed-ce3b-4c15-9b5b-33ceb4f83680","Type":"ContainerDied","Data":"0248f801e136f40acdf92bcd8c5085a4491685287bcf41f38080c8158d63d667"}
Oct 30 21:57:48.184516 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:48.184358    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0248f801e136f40acdf92bcd8c5085a4491685287bcf41f38080c8158d63d667"
Oct 30 21:57:48.928719 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:48.928693    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:57:49.081425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081383    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-kube-api-access-9hcv5\" (UniqueName: \"kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-kube-api-access-9hcv5\") pod \"63214d49-8778-4711-b6bb-6110dcc6b63b\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") "
Oct 30 21:57:49.081610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081453    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-credential-socket\") pod \"63214d49-8778-4711-b6bb-6110dcc6b63b\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") "
Oct 30 21:57:49.081610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081473    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-data\") pod \"63214d49-8778-4711-b6bb-6110dcc6b63b\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") "
Oct 30 21:57:49.081610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081519    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63214d49-8778-4711-b6bb-6110dcc6b63b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63214d49-8778-4711-b6bb-6110dcc6b63b-istiod-ca-cert\") pod \"63214d49-8778-4711-b6bb-6110dcc6b63b\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") "
Oct 30 21:57:49.081610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081559    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-socket\") pod \"63214d49-8778-4711-b6bb-6110dcc6b63b\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") "
Oct 30 21:57:49.081610 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081578    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-certs\") pod \"63214d49-8778-4711-b6bb-6110dcc6b63b\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") "
Oct 30 21:57:49.081909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081686    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-envoy\") pod \"63214d49-8778-4711-b6bb-6110dcc6b63b\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") "
Oct 30 21:57:49.081909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081750    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-token\" (UniqueName: \"kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-token\") pod \"63214d49-8778-4711-b6bb-6110dcc6b63b\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") "
Oct 30 21:57:49.081909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081758    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-credential-socket" pod "63214d49-8778-4711-b6bb-6110dcc6b63b" (UID: "63214d49-8778-4711-b6bb-6110dcc6b63b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:49.081909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081821    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-podinfo\") pod \"63214d49-8778-4711-b6bb-6110dcc6b63b\" (UID: \"63214d49-8778-4711-b6bb-6110dcc6b63b\") "
Oct 30 21:57:49.081909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081848    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-certs" pod "63214d49-8778-4711-b6bb-6110dcc6b63b" (UID: "63214d49-8778-4711-b6bb-6110dcc6b63b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:49.081909 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081860    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-socket" pod "63214d49-8778-4711-b6bb-6110dcc6b63b" (UID: "63214d49-8778-4711-b6bb-6110dcc6b63b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:49.082224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.081946    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63214d49-8778-4711-b6bb-6110dcc6b63b-istiod-ca-cert" pod "63214d49-8778-4711-b6bb-6110dcc6b63b" (UID: "63214d49-8778-4711-b6bb-6110dcc6b63b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:49.082224 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.082214    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63214d49-8778-4711-b6bb-6110dcc6b63b-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.082335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.082236    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.082578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.082367    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-data" pod "63214d49-8778-4711-b6bb-6110dcc6b63b" (UID: "63214d49-8778-4711-b6bb-6110dcc6b63b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:49.082578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.082418    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.082578 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.082444    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.083842 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.083817    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-kube-api-access-9hcv5" pod "63214d49-8778-4711-b6bb-6110dcc6b63b" (UID: "63214d49-8778-4711-b6bb-6110dcc6b63b"). InnerVolumeSpecName "kube-api-access-9hcv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:49.083992 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.083972    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-token" pod "63214d49-8778-4711-b6bb-6110dcc6b63b" (UID: "63214d49-8778-4711-b6bb-6110dcc6b63b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:49.084120 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.084100    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-envoy" pod "63214d49-8778-4711-b6bb-6110dcc6b63b" (UID: "63214d49-8778-4711-b6bb-6110dcc6b63b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:49.084190 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.084177    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-podinfo" pod "63214d49-8778-4711-b6bb-6110dcc6b63b" (UID: "63214d49-8778-4711-b6bb-6110dcc6b63b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:49.183118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.183079    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hcv5\" (UniqueName: \"kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-kube-api-access-9hcv5\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.183118 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.183118    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.183338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.183132    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.183338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.183140    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.183338 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.183149    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63214d49-8778-4711-b6bb-6110dcc6b63b-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.192382 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.192354    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"]
Oct 30 21:57:49.193725 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.193688    4918 generic.go:356] "Generic (PLEG): container finished" podID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerID="3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c" exitCode=0
Oct 30 21:57:49.193912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.193731    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" event={"ID":"63214d49-8778-4711-b6bb-6110dcc6b63b","Type":"ContainerDied","Data":"3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c"}
Oct 30 21:57:49.193912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.193762    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"
Oct 30 21:57:49.193912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.193796    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh" event={"ID":"63214d49-8778-4711-b6bb-6110dcc6b63b","Type":"ContainerDied","Data":"198d215fa54388aeaf0a03980dff8f7f3a8708eb65532a70d3d00473a22459eb"}
Oct 30 21:57:49.193912 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.193836    4918 scope.go:122] "RemoveContainer" containerID="3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c"
Oct 30 21:57:49.202714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.202691    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"]
Oct 30 21:57:49.207267 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.207249    4918 scope.go:122] "RemoveContainer" containerID="4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf"
Oct 30 21:57:49.218761 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.218736    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"]
Oct 30 21:57:49.223798 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.223775    4918 scope.go:122] "RemoveContainer" containerID="3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c"
Oct 30 21:57:49.224224 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:57:49.224193    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c\": container with ID starting with 3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c not found: ID does not exist" containerID="3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c"
Oct 30 21:57:49.224324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.224234    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c"} err="failed to get container status \"3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c\": rpc error: code = NotFound desc = could not find container \"3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c\": container with ID starting with 3a4000112d03cb121f5b0502de8060d468e92075cc6c0f5d1616fd7cf093883c not found: ID does not exist"
Oct 30 21:57:49.224324 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.224260    4918 scope.go:122] "RemoveContainer" containerID="4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf"
Oct 30 21:57:49.224813 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:57:49.224600    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf\": container with ID starting with 4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf not found: ID does not exist" containerID="4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf"
Oct 30 21:57:49.224813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.224629    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf"} err="failed to get container status \"4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf\": rpc error: code = NotFound desc = could not find container \"4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf\": container with ID starting with 4e1c98f37ae710d84ff865512fc9154c79338a1f6d5f9916b81d93f0c0356abf not found: ID does not exist"
Oct 30 21:57:49.239541 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.239518    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"]
Oct 30 21:57:49.248230 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:49.248201    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6ed261-openshift-default-74d748bc69-bscjh"]
Oct 30 21:57:51.022831 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:51.022793    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"]
Oct 30 21:57:51.025890 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:51.025865    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f831668-gdhh4"]
Oct 30 21:57:51.077104 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:51.077074    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" path="/var/lib/kubelet/pods/63214d49-8778-4711-b6bb-6110dcc6b63b/volumes"
Oct 30 21:57:51.077543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:51.077527    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c6ee490-7031-4a6d-a794-04ec2d0a360e" path="/var/lib/kubelet/pods/6c6ee490-7031-4a6d-a794-04ec2d0a360e/volumes"
Oct 30 21:57:55.230297 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.230265    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"]
Oct 30 21:57:55.230782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.230719    4918 cpu_manager.go:414] "containerMap: removing container" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy"
Oct 30 21:57:55.230782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.230736    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy"
Oct 30 21:57:55.230782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.230756    4918 cpu_manager.go:414] "containerMap: removing container" podUID="d6b795ed-ce3b-4c15-9b5b-33ceb4f83680" containerName="revoke-keys"
Oct 30 21:57:55.230782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.230762    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b795ed-ce3b-4c15-9b5b-33ceb4f83680" containerName="revoke-keys"
Oct 30 21:57:55.230782 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.230783    4918 cpu_manager.go:414] "containerMap: removing container" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy"
Oct 30 21:57:55.230954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.230788    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy"
Oct 30 21:57:55.230954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.230856    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy"
Oct 30 21:57:55.230954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.230866    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b795ed-ce3b-4c15-9b5b-33ceb4f83680" containerName="revoke-keys"
Oct 30 21:57:55.235683 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.235665    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:55.240277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.240252    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:57:55.240277 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.240263    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:57:55.243228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.243206    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lrk2v\""
Oct 30 21:57:55.243418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.243279    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:55.244265 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.244244    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"]
Oct 30 21:57:55.333321 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.333289    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b98xf\" (UniqueName: \"kubernetes.io/projected/95a87ffd-93f7-4ecd-b873-804e6b4787c1-kube-api-access-b98xf\") pod \"maas-api-revoke-sub-9bcde681-zxmlv\" (UID: \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:55.333543 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.333474    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bcde681-zxmlv\" (UID: \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:55.333603 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.333539    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bcde681-zxmlv\" (UID: \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:55.434793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.434763    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bcde681-zxmlv\" (UID: \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:55.434968 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.434812    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bcde681-zxmlv\" (UID: \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:55.435016 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.434981    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b98xf\" (UniqueName: \"kubernetes.io/projected/95a87ffd-93f7-4ecd-b873-804e6b4787c1-kube-api-access-b98xf\") pod \"maas-api-revoke-sub-9bcde681-zxmlv\" (UID: \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:55.435562 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.435493    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bcde681-zxmlv\" (UID: \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:55.437206 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.437184    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bcde681-zxmlv\" (UID: \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:55.449934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.449904    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b98xf\" (UniqueName: \"kubernetes.io/projected/95a87ffd-93f7-4ecd-b873-804e6b4787c1-kube-api-access-b98xf\") pod \"maas-api-revoke-sub-9bcde681-zxmlv\" (UID: \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:55.552998 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.552898    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:55.830077 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:55.830048    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"]
Oct 30 21:57:55.832356 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:57:55.832329    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a87ffd_93f7_4ecd_b873_804e6b4787c1.slice/crio-fa0a9a0703d0b96c73c65b8b3e7fdabbb9196b046dca311ecb1a7ce4ee3fe89d WatchSource:0}: Error finding container fa0a9a0703d0b96c73c65b8b3e7fdabbb9196b046dca311ecb1a7ce4ee3fe89d: Status 404 returned error can't find the container with id fa0a9a0703d0b96c73c65b8b3e7fdabbb9196b046dca311ecb1a7ce4ee3fe89d
Oct 30 21:57:56.242945 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:56.242911    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv" event={"ID":"95a87ffd-93f7-4ecd-b873-804e6b4787c1","Type":"ContainerStarted","Data":"fe2f26a77cd7e51e5b7a8c49bf0449e783006411837c6b1fdbca86cab0d056cb"}
Oct 30 21:57:56.243374 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:56.242960    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv" event={"ID":"95a87ffd-93f7-4ecd-b873-804e6b4787c1","Type":"ContainerStarted","Data":"fa0a9a0703d0b96c73c65b8b3e7fdabbb9196b046dca311ecb1a7ce4ee3fe89d"}
Oct 30 21:57:56.262625 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:56.262582    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv" podStartSLOduration=1.262568705 podStartE2EDuration="1.262568705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:56.260350581 +0000 UTC m=+3389.306909726" watchObservedRunningTime="2026-10-30 21:57:56.262568705 +0000 UTC m=+3389.309127849"
Oct 30 21:57:57.250425 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:57.250376    4918 generic.go:356] "Generic (PLEG): container finished" podID="95a87ffd-93f7-4ecd-b873-804e6b4787c1" containerID="fe2f26a77cd7e51e5b7a8c49bf0449e783006411837c6b1fdbca86cab0d056cb" exitCode=0
Oct 30 21:57:57.250904 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:57.250433    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv" event={"ID":"95a87ffd-93f7-4ecd-b873-804e6b4787c1","Type":"ContainerDied","Data":"fe2f26a77cd7e51e5b7a8c49bf0449e783006411837c6b1fdbca86cab0d056cb"}
Oct 30 21:57:58.394966 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:58.394940    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:58.466080 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:58.466048    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a87ffd-93f7-4ecd-b873-804e6b4787c1-kube-api-access-b98xf\" (UniqueName: \"kubernetes.io/projected/95a87ffd-93f7-4ecd-b873-804e6b4787c1-kube-api-access-b98xf\") pod \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\" (UID: \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\") "
Oct 30 21:57:58.466256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:58.466096    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-xks-ca\") pod \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\" (UID: \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\") "
Oct 30 21:57:58.466256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:58.466138    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-openshift-ca\") pod \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\" (UID: \"95a87ffd-93f7-4ecd-b873-804e6b4787c1\") "
Oct 30 21:57:58.466616 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:58.466587    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-openshift-ca" pod "95a87ffd-93f7-4ecd-b873-804e6b4787c1" (UID: "95a87ffd-93f7-4ecd-b873-804e6b4787c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:58.468315 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:58.468285    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a87ffd-93f7-4ecd-b873-804e6b4787c1-kube-api-access-b98xf" pod "95a87ffd-93f7-4ecd-b873-804e6b4787c1" (UID: "95a87ffd-93f7-4ecd-b873-804e6b4787c1"). InnerVolumeSpecName "kube-api-access-b98xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:58.468428 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:58.468417    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-xks-ca" pod "95a87ffd-93f7-4ecd-b873-804e6b4787c1" (UID: "95a87ffd-93f7-4ecd-b873-804e6b4787c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:58.567268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:58.567182    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b98xf\" (UniqueName: \"kubernetes.io/projected/95a87ffd-93f7-4ecd-b873-804e6b4787c1-kube-api-access-b98xf\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:58.567268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:58.567212    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-xks-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:58.567268 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:58.567224    4918 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95a87ffd-93f7-4ecd-b873-804e6b4787c1-maas-api-openshift-ca\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:59.159103 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:57:59.159065    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:57:59.263512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:59.263472    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv" event={"ID":"95a87ffd-93f7-4ecd-b873-804e6b4787c1","Type":"ContainerDied","Data":"fa0a9a0703d0b96c73c65b8b3e7fdabbb9196b046dca311ecb1a7ce4ee3fe89d"}
Oct 30 21:57:59.263512 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:59.263506    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcde681-zxmlv"
Oct 30 21:57:59.263731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:57:59.263538    4918 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa0a9a0703d0b96c73c65b8b3e7fdabbb9196b046dca311ecb1a7ce4ee3fe89d"
Oct 30 21:58:03.023431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:03.023384    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"]
Oct 30 21:58:03.025923 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:03.025897    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9c0bcd-vmkc8"]
Oct 30 21:58:03.077180 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:03.077151    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe86e610-3b67-487f-9a62-8355e7de6cd1" path="/var/lib/kubelet/pods/fe86e610-3b67-487f-9a62-8355e7de6cd1/volumes"
Oct 30 21:58:04.217319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.217285    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main" containerID="cri-o://614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7" gracePeriod=45
Oct 30 21:58:04.228843 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.228812    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main" containerID="cri-o://cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed" gracePeriod=45
Oct 30 21:58:04.242509 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.242468    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main" containerID="cri-o://5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed" gracePeriod=45
Oct 30 21:58:04.519858 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.519822    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:58:04.533622 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.533599    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:58:04.536688 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.536670    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:58:04.625822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.625785    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-model-cache\") pod \"970c2613-6a80-4b60-bfce-a51393982944\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") "
Oct 30 21:58:04.626045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.625858    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-home\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-home\") pod \"970c2613-6a80-4b60-bfce-a51393982944\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") "
Oct 30 21:58:04.626045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.625885    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-tmp-dir\") pod \"970c2613-6a80-4b60-bfce-a51393982944\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") "
Oct 30 21:58:04.626045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.625920    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-dshm\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-dshm\") pod \"970c2613-6a80-4b60-bfce-a51393982944\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") "
Oct 30 21:58:04.626045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.625979    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3279db-e5d3-4bce-bf19-490161f761a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c3279db-e5d3-4bce-bf19-490161f761a2-tls-certs\") pod \"8c3279db-e5d3-4bce-bf19-490161f761a2\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") "
Oct 30 21:58:04.626045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626009    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-dshm\") pod \"2ebb1491-7850-4c93-8087-6d325f8f32c2\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") "
Oct 30 21:58:04.626045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626041    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-tmp-dir\") pod \"8c3279db-e5d3-4bce-bf19-490161f761a2\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") "
Oct 30 21:58:04.626490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626069    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3279db-e5d3-4bce-bf19-490161f761a2-kube-api-access-mbnrq\" (UniqueName: \"kubernetes.io/projected/8c3279db-e5d3-4bce-bf19-490161f761a2-kube-api-access-mbnrq\") pod \"8c3279db-e5d3-4bce-bf19-490161f761a2\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") "
Oct 30 21:58:04.626490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626107    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-home\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-home\") pod \"8c3279db-e5d3-4bce-bf19-490161f761a2\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") "
Oct 30 21:58:04.626490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626132    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-home\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-home\") pod \"2ebb1491-7850-4c93-8087-6d325f8f32c2\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") "
Oct 30 21:58:04.626490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626119    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-model-cache" pod "970c2613-6a80-4b60-bfce-a51393982944" (UID: "970c2613-6a80-4b60-bfce-a51393982944"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.626490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626161    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-model-cache\") pod \"8c3279db-e5d3-4bce-bf19-490161f761a2\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") "
Oct 30 21:58:04.626490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626234    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-tmp-dir" pod "970c2613-6a80-4b60-bfce-a51393982944" (UID: "970c2613-6a80-4b60-bfce-a51393982944"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.626490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626330    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-tmp-dir" pod "8c3279db-e5d3-4bce-bf19-490161f761a2" (UID: "8c3279db-e5d3-4bce-bf19-490161f761a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.626490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626346    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-home" pod "970c2613-6a80-4b60-bfce-a51393982944" (UID: "970c2613-6a80-4b60-bfce-a51393982944"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.626490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626376    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ebb1491-7850-4c93-8087-6d325f8f32c2-kube-api-access-kfp6r\" (UniqueName: \"kubernetes.io/projected/2ebb1491-7850-4c93-8087-6d325f8f32c2-kube-api-access-kfp6r\") pod \"2ebb1491-7850-4c93-8087-6d325f8f32c2\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") "
Oct 30 21:58:04.626490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626424    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/970c2613-6a80-4b60-bfce-a51393982944-tls-certs\" (UniqueName: \"kubernetes.io/secret/970c2613-6a80-4b60-bfce-a51393982944-tls-certs\") pod \"970c2613-6a80-4b60-bfce-a51393982944\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") "
Oct 30 21:58:04.626490 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626466    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-dshm\") pod \"8c3279db-e5d3-4bce-bf19-490161f761a2\" (UID: \"8c3279db-e5d3-4bce-bf19-490161f761a2\") "
Oct 30 21:58:04.627184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626513    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/970c2613-6a80-4b60-bfce-a51393982944-kube-api-access-6xcd4\" (UniqueName: \"kubernetes.io/projected/970c2613-6a80-4b60-bfce-a51393982944-kube-api-access-6xcd4\") pod \"970c2613-6a80-4b60-bfce-a51393982944\" (UID: \"970c2613-6a80-4b60-bfce-a51393982944\") "
Oct 30 21:58:04.627184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626555    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-model-cache\") pod \"2ebb1491-7850-4c93-8087-6d325f8f32c2\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") "
Oct 30 21:58:04.627184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626565    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-home" pod "2ebb1491-7850-4c93-8087-6d325f8f32c2" (UID: "2ebb1491-7850-4c93-8087-6d325f8f32c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.627184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626583    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ebb1491-7850-4c93-8087-6d325f8f32c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebb1491-7850-4c93-8087-6d325f8f32c2-tls-certs\") pod \"2ebb1491-7850-4c93-8087-6d325f8f32c2\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") "
Oct 30 21:58:04.627184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626615    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-tmp-dir\") pod \"2ebb1491-7850-4c93-8087-6d325f8f32c2\" (UID: \"2ebb1491-7850-4c93-8087-6d325f8f32c2\") "
Oct 30 21:58:04.627184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626924    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-home" pod "8c3279db-e5d3-4bce-bf19-490161f761a2" (UID: "8c3279db-e5d3-4bce-bf19-490161f761a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.627184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.626956    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-model-cache" pod "2ebb1491-7850-4c93-8087-6d325f8f32c2" (UID: "2ebb1491-7850-4c93-8087-6d325f8f32c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.627184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.627032    4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.627184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.627053    4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.627184 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.627130    4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.627893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.627228    4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.627893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.627285    4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.627893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.627298    4918 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-home\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.627893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.627520    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-tmp-dir" pod "2ebb1491-7850-4c93-8087-6d325f8f32c2" (UID: "2ebb1491-7850-4c93-8087-6d325f8f32c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.627893 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.627694    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-model-cache" pod "8c3279db-e5d3-4bce-bf19-490161f761a2" (UID: "8c3279db-e5d3-4bce-bf19-490161f761a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.630556 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.630526    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ebb1491-7850-4c93-8087-6d325f8f32c2-tls-certs" pod "2ebb1491-7850-4c93-8087-6d325f8f32c2" (UID: "2ebb1491-7850-4c93-8087-6d325f8f32c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:04.630699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.630599    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3279db-e5d3-4bce-bf19-490161f761a2-tls-certs" pod "8c3279db-e5d3-4bce-bf19-490161f761a2" (UID: "8c3279db-e5d3-4bce-bf19-490161f761a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:04.630699 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.630649    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-dshm" pod "8c3279db-e5d3-4bce-bf19-490161f761a2" (UID: "8c3279db-e5d3-4bce-bf19-490161f761a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.631003 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.630975    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/970c2613-6a80-4b60-bfce-a51393982944-tls-certs" pod "970c2613-6a80-4b60-bfce-a51393982944" (UID: "970c2613-6a80-4b60-bfce-a51393982944"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:04.631121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.631053    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3279db-e5d3-4bce-bf19-490161f761a2-kube-api-access-mbnrq" pod "8c3279db-e5d3-4bce-bf19-490161f761a2" (UID: "8c3279db-e5d3-4bce-bf19-490161f761a2"). InnerVolumeSpecName "kube-api-access-mbnrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:04.631413 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.631358    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ebb1491-7850-4c93-8087-6d325f8f32c2-kube-api-access-kfp6r" pod "2ebb1491-7850-4c93-8087-6d325f8f32c2" (UID: "2ebb1491-7850-4c93-8087-6d325f8f32c2"). InnerVolumeSpecName "kube-api-access-kfp6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:04.631545 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.631527    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970c2613-6a80-4b60-bfce-a51393982944-kube-api-access-6xcd4" pod "970c2613-6a80-4b60-bfce-a51393982944" (UID: "970c2613-6a80-4b60-bfce-a51393982944"). InnerVolumeSpecName "kube-api-access-6xcd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:04.631822 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.631783    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-dshm" pod "2ebb1491-7850-4c93-8087-6d325f8f32c2" (UID: "2ebb1491-7850-4c93-8087-6d325f8f32c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.632121 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.632102    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-dshm" pod "970c2613-6a80-4b60-bfce-a51393982944" (UID: "970c2613-6a80-4b60-bfce-a51393982944"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:04.728344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.728309    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfp6r\" (UniqueName: \"kubernetes.io/projected/2ebb1491-7850-4c93-8087-6d325f8f32c2-kube-api-access-kfp6r\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.728344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.728340    4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/970c2613-6a80-4b60-bfce-a51393982944-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.728344 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.728350    4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.728638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.728360    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xcd4\" (UniqueName: \"kubernetes.io/projected/970c2613-6a80-4b60-bfce-a51393982944-kube-api-access-6xcd4\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.728638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.728369    4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.728638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.728379    4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebb1491-7850-4c93-8087-6d325f8f32c2-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.728638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.728407    4918 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-tmp-dir\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.728638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.728416    4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/970c2613-6a80-4b60-bfce-a51393982944-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.728638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.728424    4918 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c3279db-e5d3-4bce-bf19-490161f761a2-tls-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.728638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.728431    4918 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebb1491-7850-4c93-8087-6d325f8f32c2-dshm\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.728638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.728438    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbnrq\" (UniqueName: \"kubernetes.io/projected/8c3279db-e5d3-4bce-bf19-490161f761a2-kube-api-access-mbnrq\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:04.728638 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:04.728446    4918 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c3279db-e5d3-4bce-bf19-490161f761a2-model-cache\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:05.301927 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.301891    4918 generic.go:356] "Generic (PLEG): container finished" podID="970c2613-6a80-4b60-bfce-a51393982944" containerID="614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7" exitCode=0
Oct 30 21:58:05.302432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.301936    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"
Oct 30 21:58:05.302432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.301979    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" event={"ID":"970c2613-6a80-4b60-bfce-a51393982944","Type":"ContainerDied","Data":"614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7"}
Oct 30 21:58:05.302432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.302028    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx" event={"ID":"970c2613-6a80-4b60-bfce-a51393982944","Type":"ContainerDied","Data":"01a653659c1c367628fbb1d0dace610aa0d0fea190af228b092efe5470d933f2"}
Oct 30 21:58:05.302432 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.302053    4918 scope.go:122] "RemoveContainer" containerID="614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7"
Oct 30 21:58:05.303819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.303772    4918 generic.go:356] "Generic (PLEG): container finished" podID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerID="5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed" exitCode=0
Oct 30 21:58:05.303819 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.303816    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"
Oct 30 21:58:05.304008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.303856    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" event={"ID":"8c3279db-e5d3-4bce-bf19-490161f761a2","Type":"ContainerDied","Data":"5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed"}
Oct 30 21:58:05.304008 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.303894    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg" event={"ID":"8c3279db-e5d3-4bce-bf19-490161f761a2","Type":"ContainerDied","Data":"d5e30a29b94ee846cf6808f1d53a3c781ad545b80883b7cb3bf4c66ac7bcad1c"}
Oct 30 21:58:05.305572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.305551    4918 generic.go:356] "Generic (PLEG): container finished" podID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerID="cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed" exitCode=0
Oct 30 21:58:05.305714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.305582    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" event={"ID":"2ebb1491-7850-4c93-8087-6d325f8f32c2","Type":"ContainerDied","Data":"cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed"}
Oct 30 21:58:05.305714 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.305601    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"
Oct 30 21:58:05.305860 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.305608    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw" event={"ID":"2ebb1491-7850-4c93-8087-6d325f8f32c2","Type":"ContainerDied","Data":"fac5c9d3c289d7d563cb49e9162053dbe78cac9a701134d49afba00eae2412c1"}
Oct 30 21:58:05.313011 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.312993    4918 scope.go:122] "RemoveContainer" containerID="614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7"
Oct 30 21:58:05.313261 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:58:05.313242    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7\": container with ID starting with 614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7 not found: ID does not exist" containerID="614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7"
Oct 30 21:58:05.313309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.313271    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7"} err="failed to get container status \"614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7\": rpc error: code = NotFound desc = could not find container \"614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7\": container with ID starting with 614a7b7bfed6afe89d6d2bbc1624d8f10aedd21407c33745e45e57907f15dee7 not found: ID does not exist"
Oct 30 21:58:05.313309 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.313287    4918 scope.go:122] "RemoveContainer" containerID="5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed"
Oct 30 21:58:05.324022 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.324009    4918 scope.go:122] "RemoveContainer" containerID="5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed"
Oct 30 21:58:05.324452 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:58:05.324425    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed\": container with ID starting with 5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed not found: ID does not exist" containerID="5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed"
Oct 30 21:58:05.324621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.324462    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed"} err="failed to get container status \"5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed\": rpc error: code = NotFound desc = could not find container \"5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed\": container with ID starting with 5fc028e99a74657d4dad810dd401d827686909497b9a28312e2e261f363523ed not found: ID does not exist"
Oct 30 21:58:05.324621 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.324486    4918 scope.go:122] "RemoveContainer" containerID="cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed"
Oct 30 21:58:05.327187 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.327169    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"]
Oct 30 21:58:05.331587 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.331565    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/simulator-w2-05a342-kserve-5864486c54-tspkg"]
Oct 30 21:58:05.334027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.333988    4918 scope.go:122] "RemoveContainer" containerID="cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed"
Oct 30 21:58:05.334315 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:58:05.334297    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed\": container with ID starting with cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed not found: ID does not exist" containerID="cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed"
Oct 30 21:58:05.334362 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.334324    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed"} err="failed to get container status \"cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed\": rpc error: code = NotFound desc = could not find container \"cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed\": container with ID starting with cdbfea9d7cf3e4952f6eea418936846b82e83eac27edac60a0bbb061b80594ed not found: ID does not exist"
Oct 30 21:58:05.344070 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.344049    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"]
Oct 30 21:58:05.349045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.349026    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/distinct-w2-05a342-kserve-7cc666b998-dsnpx"]
Oct 30 21:58:05.362436 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.362416    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"]
Oct 30 21:58:05.368057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:05.368035    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-05a342/premium-w2-05a342-kserve-9d8b876f5-766vw"]
Oct 30 21:58:07.075662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:07.075634    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" path="/var/lib/kubelet/pods/2ebb1491-7850-4c93-8087-6d325f8f32c2/volumes"
Oct 30 21:58:07.076076 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:07.076061    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" path="/var/lib/kubelet/pods/8c3279db-e5d3-4bce-bf19-490161f761a2/volumes"
Oct 30 21:58:07.076446 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:07.076431    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="970c2613-6a80-4b60-bfce-a51393982944" path="/var/lib/kubelet/pods/970c2613-6a80-4b60-bfce-a51393982944/volumes"
Oct 30 21:58:10.644068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:10.644028    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"]
Oct 30 21:58:10.690373 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:10.690337    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-05a342-vkwwl"]
Oct 30 21:58:11.075233 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:11.075201    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b795ed-ce3b-4c15-9b5b-33ceb4f83680" path="/var/lib/kubelet/pods/d6b795ed-ce3b-4c15-9b5b-33ceb4f83680/volumes"
Oct 30 21:58:17.622770 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:17.622736    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"]
Oct 30 21:58:17.623231 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:17.622922    4918 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2" podUID="c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" containerName="istio-proxy" containerID="cri-o://29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b" gracePeriod=30
Oct 30 21:58:22.023310 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:22.023276    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"]
Oct 30 21:58:22.025576 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:22.025556    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c13475c-9sfnr"]
Oct 30 21:58:22.873732 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:22.873705    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:58:23.007777 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.007743    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-envoy\") pod \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") "
Oct 30 21:58:23.007993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.007792    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-credential-socket\") pod \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") "
Oct 30 21:58:23.007993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.007825    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-kube-api-access-fv86p\" (UniqueName: \"kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-kube-api-access-fv86p\") pod \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") "
Oct 30 21:58:23.007993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.007864    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-certs\") pod \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") "
Oct 30 21:58:23.007993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.007905    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-socket\") pod \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") "
Oct 30 21:58:23.007993 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.007934    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-data\") pod \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") "
Oct 30 21:58:23.008275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.008052    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-token\" (UniqueName: \"kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-token\") pod \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") "
Oct 30 21:58:23.008275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.008135    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-podinfo\") pod \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") "
Oct 30 21:58:23.008275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.008140    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-credential-socket" pod "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" (UID: "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:23.008275 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.008175    4918 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istiod-ca-cert\") pod \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\" (UID: \"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5\") "
Oct 30 21:58:23.008555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.008261    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-socket" pod "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" (UID: "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:23.008555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.008274    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-certs" pod "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" (UID: "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:23.008555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.008478    4918 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-credential-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.008555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.008540    4918 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-certs\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.008555 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.008556    4918 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-workload-socket\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.008800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.008573    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-data" pod "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" (UID: "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:23.008800 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.008667    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istiod-ca-cert" pod "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" (UID: "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:23.010202 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.010170    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-envoy" pod "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" (UID: "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:23.010429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.010407    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-token" pod "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" (UID: "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:23.010429 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.010384    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-kube-api-access-fv86p" pod "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" (UID: "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5"). InnerVolumeSpecName "kube-api-access-fv86p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:23.010641 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.010621    4918 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-podinfo" pod "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" (UID: "c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:58:23.075849 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.075819    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7dad876-777d-4252-b6ec-3cc41283b6df" path="/var/lib/kubelet/pods/a7dad876-777d-4252-b6ec-3cc41283b6df/volumes"
Oct 30 21:58:23.109434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.109382    4918 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv86p\" (UniqueName: \"kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-kube-api-access-fv86p\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.109434 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.109431    4918 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-data\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.109595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.109447    4918 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-token\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.109595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.109460    4918 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-podinfo\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.109595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.109472    4918 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istiod-ca-cert\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.109595 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.109485    4918 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5-istio-envoy\") on node \"ip-10-0-139-90.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.427027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.426939    4918 generic.go:356] "Generic (PLEG): container finished" podID="c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" containerID="29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b" exitCode=0
Oct 30 21:58:23.427027 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.426987    4918 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"
Oct 30 21:58:23.427243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.427025    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2" event={"ID":"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5","Type":"ContainerDied","Data":"29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b"}
Oct 30 21:58:23.427243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.427087    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2" event={"ID":"c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5","Type":"ContainerDied","Data":"8a92c5bf5d77f426581e4f524c33af1b638471b6f297a86002c34799a4fc7723"}
Oct 30 21:58:23.427243 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.427112    4918 scope.go:122] "RemoveContainer" containerID="29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b"
Oct 30 21:58:23.437786 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.437767    4918 scope.go:122] "RemoveContainer" containerID="29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b"
Oct 30 21:58:23.438075 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:58:23.438054    4918 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b\": container with ID starting with 29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b not found: ID does not exist" containerID="29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b"
Oct 30 21:58:23.438145 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.438088    4918 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b"} err="failed to get container status \"29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b\": rpc error: code = NotFound desc = could not find container \"29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b\": container with ID starting with 29c02bf9134d7eb41bc28e40d7b5b7c1e9a5fc85e246ba84499056b599d22e2b not found: ID does not exist"
Oct 30 21:58:23.448200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.448178    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"]
Oct 30 21:58:23.453583 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:23.453565    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-05a342-openshift-default-6cd998c4ff-xj7d2"]
Oct 30 21:58:25.075043 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:25.075011    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" path="/var/lib/kubelet/pods/c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5/volumes"
Oct 30 21:58:29.159408 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:58:29.159351    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:58:33.022458 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:33.022419    4918 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"]
Oct 30 21:58:33.025454 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:33.025428    4918 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84370ca7-6prtl"]
Oct 30 21:58:33.075407 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:33.075350    4918 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf29a31-829a-45a0-85a6-1549139db5c5" path="/var/lib/kubelet/pods/1bf29a31-829a-45a0-85a6-1549139db5c5/volumes"
Oct 30 21:58:35.665418 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:35.665365    4918 scope.go:122] "RemoveContainer" containerID="960472a07ddb7f982786003e6ef9c964e68b6e88bce2778d67da93fd9a4aec60"
Oct 30 21:58:35.675069 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:35.675048    4918 scope.go:122] "RemoveContainer" containerID="df22001c9b868b35d9190a173fc7d0f82888e8ed028e5578e3d5a028d17653e5"
Oct 30 21:58:35.683646 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:35.683627    4918 scope.go:122] "RemoveContainer" containerID="e04d1df95940f40b66568bc5a56841d8ce5439098f0d5601e1d34e1dae8283e8"
Oct 30 21:58:36.233253 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:36.233224    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1c5a4a64-lvmch_d2f9f82a-e844-44ff-8bb9-87cbaabb8dc3/revoke-keys/0.log"
Oct 30 21:58:36.499349 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:36.499270    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-22d45c89-tmzfv_34140bfe-9c3d-4a45-a512-000a2702ae94/revoke-keys/0.log"
Oct 30 21:58:36.667046 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:36.667016    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-379bb5b6-l6d4p_b209119c-222d-444f-a597-43fdc39d42ac/revoke-keys/0.log"
Oct 30 21:58:36.846763 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:36.846690    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-93653586-m58wl_9b0e04ce-7e47-4e75-afb8-b0c06c7e6cec/revoke-keys/0.log"
Oct 30 21:58:37.019181 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:37.019153    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9bcde681-zxmlv_95a87ffd-93f7-4ecd-b873-804e6b4787c1/revoke-keys/0.log"
Oct 30 21:58:37.193102 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:37.193019    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a3d9a021-cddb9_fa1f06a3-7b8e-47cb-b7d4-418250d4f2bf/revoke-keys/0.log"
Oct 30 21:58:37.359585 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:37.359554    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c61ec39f-k8wcz_6b887953-b45a-4eb9-808c-dc5974efda1e/revoke-keys/0.log"
Oct 30 21:58:37.698228 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:37.698197    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cf82b429-vjqtw_e258e5ec-224a-4747-9880-0b09bb19265a/revoke-keys/0.log"
Oct 30 21:58:37.871007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:37.870974    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fc834391-hh5m7_3b6d36a7-18cf-4223-8c47-e199ecf31b3c/revoke-keys/0.log"
Oct 30 21:58:42.305679 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:42.305650    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-fb99t_9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1/discovery/0.log"
Oct 30 21:58:42.494116 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:42.494089    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5978974d9d-92d2b_7971b944-89b9-447c-96d8-8111a389680b/kube-auth-proxy/0.log"
Oct 30 21:58:43.211473 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:43.211437    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-d54n7_40508368-7d69-41e3-a2ee-0be190a20281/istio-proxy/0.log"
Oct 30 21:58:43.889023 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:43.888991    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-w9kjp_de5f2249-cc22-496c-b84a-8d1fcf7c5fc7/istio-proxy/0.log"
Oct 30 21:58:44.887571 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:44.887534    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d5fbcf987-vhd7f_7c27677e-bbe2-4457-b205-1faae8635cb0/router/0.log"
Oct 30 21:58:45.851970 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:45.851938    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-xmt8x_2825e8f7-cceb-46f1-87c9-23d30491e8b3/main/0.log"
Oct 30 21:58:46.368032 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:46.368000    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-hmt8h_4b55befd-378d-4d0c-9d54-d9f500c08c22/main/0.log"
Oct 30 21:58:46.905280 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:46.905245    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-cztkf_83a3d45f-aa19-4b00-b461-2896fe5929f6/main/0.log"
Oct 30 21:58:47.095341 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:47.095307    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-j5xdq_1ec8e779-955f-4570-81ad-480d8cefca7a/main/0.log"
Oct 30 21:58:57.498627 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.498587    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vjht4/must-gather-jmmcm"]
Oct 30 21:58:57.499035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.498966    4918 cpu_manager.go:414] "containerMap: removing container" podUID="c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" containerName="istio-proxy"
Oct 30 21:58:57.499035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.498978    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" containerName="istio-proxy"
Oct 30 21:58:57.499035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499000    4918 cpu_manager.go:414] "containerMap: removing container" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main"
Oct 30 21:58:57.499035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499006    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main"
Oct 30 21:58:57.499035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499013    4918 cpu_manager.go:414] "containerMap: removing container" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main"
Oct 30 21:58:57.499035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499020    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main"
Oct 30 21:58:57.499256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499053    4918 cpu_manager.go:414] "containerMap: removing container" podUID="95a87ffd-93f7-4ecd-b873-804e6b4787c1" containerName="revoke-keys"
Oct 30 21:58:57.499256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499059    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a87ffd-93f7-4ecd-b873-804e6b4787c1" containerName="revoke-keys"
Oct 30 21:58:57.499256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499077    4918 cpu_manager.go:414] "containerMap: removing container" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main"
Oct 30 21:58:57.499256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499082    4918 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main"
Oct 30 21:58:57.499256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499135    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a87ffd-93f7-4ecd-b873-804e6b4787c1" containerName="revoke-keys"
Oct 30 21:58:57.499256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499147    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="63214d49-8778-4711-b6bb-6110dcc6b63b" containerName="istio-proxy"
Oct 30 21:58:57.499256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499157    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3279db-e5d3-4bce-bf19-490161f761a2" containerName="main"
Oct 30 21:58:57.499256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499164    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a6cc5b-be20-4ad9-ab5b-e60852ee21b5" containerName="istio-proxy"
Oct 30 21:58:57.499256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499171    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ebb1491-7850-4c93-8087-6d325f8f32c2" containerName="main"
Oct 30 21:58:57.499256 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.499177    4918 memory_manager.go:365] "RemoveStaleState removing state" podUID="970c2613-6a80-4b60-bfce-a51393982944" containerName="main"
Oct 30 21:58:57.503990 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.503972    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vjht4/must-gather-jmmcm"
Oct 30 21:58:57.513677 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.513652    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vjht4\"/\"default-dockercfg-nvn77\""
Oct 30 21:58:57.513793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.513732    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vjht4\"/\"kube-root-ca.crt\""
Oct 30 21:58:57.513793 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.513742    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vjht4\"/\"openshift-service-ca.crt\""
Oct 30 21:58:57.514449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.514429    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vjht4/must-gather-jmmcm"]
Oct 30 21:58:57.585132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.585102    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qm88q/must-gather-vgsm7"]
Oct 30 21:58:57.589086 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.589065    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qzjln/must-gather-rrkrz"]
Oct 30 21:58:57.589219 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.589195    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qm88q/must-gather-vgsm7"
Oct 30 21:58:57.601977 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.601952    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzjln/must-gather-rrkrz"
Oct 30 21:58:57.605377 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.605352    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qm88q/must-gather-vgsm7"]
Oct 30 21:58:57.607200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.607178    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzjln/must-gather-rrkrz"]
Oct 30 21:58:57.610135 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.610108    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5-must-gather-output\") pod \"must-gather-jmmcm\" (UID: \"2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5\") " pod="openshift-must-gather-vjht4/must-gather-jmmcm"
Oct 30 21:58:57.610368 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.610335    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d7ll\" (UniqueName: \"kubernetes.io/projected/2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5-kube-api-access-9d7ll\") pod \"must-gather-jmmcm\" (UID: \"2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5\") " pod="openshift-must-gather-vjht4/must-gather-jmmcm"
Oct 30 21:58:57.611939 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.611917    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qzjln\"/\"default-dockercfg-cbx5l\""
Oct 30 21:58:57.612043 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.612029    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qzjln\"/\"kube-root-ca.crt\""
Oct 30 21:58:57.612110 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.612103    4918 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qm88q\"/\"default-dockercfg-k7585\""
Oct 30 21:58:57.612201 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.612188    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qm88q\"/\"openshift-service-ca.crt\""
Oct 30 21:58:57.612330 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.612317    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qm88q\"/\"kube-root-ca.crt\""
Oct 30 21:58:57.612369 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.612336    4918 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qzjln\"/\"openshift-service-ca.crt\""
Oct 30 21:58:57.711200 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.711166    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50e0c8d1-3fdb-45c6-b752-f073f276a44c-must-gather-output\") pod \"must-gather-rrkrz\" (UID: \"50e0c8d1-3fdb-45c6-b752-f073f276a44c\") " pod="openshift-must-gather-qzjln/must-gather-rrkrz"
Oct 30 21:58:57.711464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.711216    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19a277e1-4828-4800-a5b9-d514cd95bc94-must-gather-output\") pod \"must-gather-vgsm7\" (UID: \"19a277e1-4828-4800-a5b9-d514cd95bc94\") " pod="openshift-must-gather-qm88q/must-gather-vgsm7"
Oct 30 21:58:57.711464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.711238    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc9sv\" (UniqueName: \"kubernetes.io/projected/50e0c8d1-3fdb-45c6-b752-f073f276a44c-kube-api-access-cc9sv\") pod \"must-gather-rrkrz\" (UID: \"50e0c8d1-3fdb-45c6-b752-f073f276a44c\") " pod="openshift-must-gather-qzjln/must-gather-rrkrz"
Oct 30 21:58:57.711464 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.711380    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5-must-gather-output\") pod \"must-gather-jmmcm\" (UID: \"2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5\") " pod="openshift-must-gather-vjht4/must-gather-jmmcm"
Oct 30 21:58:57.711671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.711468    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tb72\" (UniqueName: \"kubernetes.io/projected/19a277e1-4828-4800-a5b9-d514cd95bc94-kube-api-access-4tb72\") pod \"must-gather-vgsm7\" (UID: \"19a277e1-4828-4800-a5b9-d514cd95bc94\") " pod="openshift-must-gather-qm88q/must-gather-vgsm7"
Oct 30 21:58:57.711671 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.711518    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d7ll\" (UniqueName: \"kubernetes.io/projected/2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5-kube-api-access-9d7ll\") pod \"must-gather-jmmcm\" (UID: \"2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5\") " pod="openshift-must-gather-vjht4/must-gather-jmmcm"
Oct 30 21:58:57.711789 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.711704    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5-must-gather-output\") pod \"must-gather-jmmcm\" (UID: \"2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5\") " pod="openshift-must-gather-vjht4/must-gather-jmmcm"
Oct 30 21:58:57.720655 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.720635    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d7ll\" (UniqueName: \"kubernetes.io/projected/2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5-kube-api-access-9d7ll\") pod \"must-gather-jmmcm\" (UID: \"2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5\") " pod="openshift-must-gather-vjht4/must-gather-jmmcm"
Oct 30 21:58:57.812609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.812519    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50e0c8d1-3fdb-45c6-b752-f073f276a44c-must-gather-output\") pod \"must-gather-rrkrz\" (UID: \"50e0c8d1-3fdb-45c6-b752-f073f276a44c\") " pod="openshift-must-gather-qzjln/must-gather-rrkrz"
Oct 30 21:58:57.812609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.812579    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19a277e1-4828-4800-a5b9-d514cd95bc94-must-gather-output\") pod \"must-gather-vgsm7\" (UID: \"19a277e1-4828-4800-a5b9-d514cd95bc94\") " pod="openshift-must-gather-qm88q/must-gather-vgsm7"
Oct 30 21:58:57.812609 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.812599    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc9sv\" (UniqueName: \"kubernetes.io/projected/50e0c8d1-3fdb-45c6-b752-f073f276a44c-kube-api-access-cc9sv\") pod \"must-gather-rrkrz\" (UID: \"50e0c8d1-3fdb-45c6-b752-f073f276a44c\") " pod="openshift-must-gather-qzjln/must-gather-rrkrz"
Oct 30 21:58:57.812869 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.812649    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tb72\" (UniqueName: \"kubernetes.io/projected/19a277e1-4828-4800-a5b9-d514cd95bc94-kube-api-access-4tb72\") pod \"must-gather-vgsm7\" (UID: \"19a277e1-4828-4800-a5b9-d514cd95bc94\") " pod="openshift-must-gather-qm88q/must-gather-vgsm7"
Oct 30 21:58:57.812967 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.812944    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19a277e1-4828-4800-a5b9-d514cd95bc94-must-gather-output\") pod \"must-gather-vgsm7\" (UID: \"19a277e1-4828-4800-a5b9-d514cd95bc94\") " pod="openshift-must-gather-qm88q/must-gather-vgsm7"
Oct 30 21:58:57.813045 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.812944    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50e0c8d1-3fdb-45c6-b752-f073f276a44c-must-gather-output\") pod \"must-gather-rrkrz\" (UID: \"50e0c8d1-3fdb-45c6-b752-f073f276a44c\") " pod="openshift-must-gather-qzjln/must-gather-rrkrz"
Oct 30 21:58:57.819170 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.819149    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vjht4/must-gather-jmmcm"
Oct 30 21:58:57.821731 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.821710    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc9sv\" (UniqueName: \"kubernetes.io/projected/50e0c8d1-3fdb-45c6-b752-f073f276a44c-kube-api-access-cc9sv\") pod \"must-gather-rrkrz\" (UID: \"50e0c8d1-3fdb-45c6-b752-f073f276a44c\") " pod="openshift-must-gather-qzjln/must-gather-rrkrz"
Oct 30 21:58:57.821830 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.821789    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tb72\" (UniqueName: \"kubernetes.io/projected/19a277e1-4828-4800-a5b9-d514cd95bc94-kube-api-access-4tb72\") pod \"must-gather-vgsm7\" (UID: \"19a277e1-4828-4800-a5b9-d514cd95bc94\") " pod="openshift-must-gather-qm88q/must-gather-vgsm7"
Oct 30 21:58:57.911358 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.911332    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qm88q/must-gather-vgsm7"
Oct 30 21:58:57.918449 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:57.918433    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzjln/must-gather-rrkrz"
Oct 30 21:58:58.099813 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:58.099788    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vjht4/must-gather-jmmcm"]
Oct 30 21:58:58.101524 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:58:58.101499    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fabd8a6_faae_4e27_a2df_12c8b3d8f2d5.slice/crio-95f5885cb96d865b049b93763e94d728dd837ae011ceb6f594c9cf55cff62d73 WatchSource:0}: Error finding container 95f5885cb96d865b049b93763e94d728dd837ae011ceb6f594c9cf55cff62d73: Status 404 returned error can't find the container with id 95f5885cb96d865b049b93763e94d728dd837ae011ceb6f594c9cf55cff62d73
Oct 30 21:58:58.103319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:58.103302    4918 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:58:58.189580 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:58.189550    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qm88q/must-gather-vgsm7"]
Oct 30 21:58:58.191416 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:58:58.191372    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19a277e1_4828_4800_a5b9_d514cd95bc94.slice/crio-727a61b98c8454076d27b8c6398043eb3d05326a4d63d2cccf3193bdf659ef35 WatchSource:0}: Error finding container 727a61b98c8454076d27b8c6398043eb3d05326a4d63d2cccf3193bdf659ef35: Status 404 returned error can't find the container with id 727a61b98c8454076d27b8c6398043eb3d05326a4d63d2cccf3193bdf659ef35
Oct 30 21:58:58.200599 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:58.200578    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzjln/must-gather-rrkrz"]
Oct 30 21:58:58.203062 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:58:58.203039    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50e0c8d1_3fdb_45c6_b752_f073f276a44c.slice/crio-e0f9dff06bdb1181a9ba1bbda887f1de58e38d7dc02696f9ffd2917c233efd93 WatchSource:0}: Error finding container e0f9dff06bdb1181a9ba1bbda887f1de58e38d7dc02696f9ffd2917c233efd93: Status 404 returned error can't find the container with id e0f9dff06bdb1181a9ba1bbda887f1de58e38d7dc02696f9ffd2917c233efd93
Oct 30 21:58:58.659856 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:58.659822    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qm88q/must-gather-vgsm7" event={"ID":"19a277e1-4828-4800-a5b9-d514cd95bc94","Type":"ContainerStarted","Data":"727a61b98c8454076d27b8c6398043eb3d05326a4d63d2cccf3193bdf659ef35"}
Oct 30 21:58:58.661139 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:58.661109    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzjln/must-gather-rrkrz" event={"ID":"50e0c8d1-3fdb-45c6-b752-f073f276a44c","Type":"ContainerStarted","Data":"e0f9dff06bdb1181a9ba1bbda887f1de58e38d7dc02696f9ffd2917c233efd93"}
Oct 30 21:58:58.662347 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:58.662324    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vjht4/must-gather-jmmcm" event={"ID":"2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5","Type":"ContainerStarted","Data":"95f5885cb96d865b049b93763e94d728dd837ae011ceb6f594c9cf55cff62d73"}
Oct 30 21:58:59.171633 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:58:59.171601    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:58:59.694718 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:59.694675    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzjln/must-gather-rrkrz" event={"ID":"50e0c8d1-3fdb-45c6-b752-f073f276a44c","Type":"ContainerStarted","Data":"38958ab4659eb4f7dbc8315734274e7028e22d48a398462e9c176ba2a56d82d3"}
Oct 30 21:58:59.695319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:59.694737    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzjln/must-gather-rrkrz" event={"ID":"50e0c8d1-3fdb-45c6-b752-f073f276a44c","Type":"ContainerStarted","Data":"a19d87ce135890cda507fcff2374820acffa1746f3bc399df0b5cf83d45d772f"}
Oct 30 21:58:59.716035 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:58:59.715091    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qzjln/must-gather-rrkrz" podStartSLOduration=1.9813911690000001 podStartE2EDuration="2.715070265s" totalImagesPullingTime="733.679096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:59.713780636 +0000 UTC m=+3452.760339782" watchObservedRunningTime="2026-10-30 21:58:59.715070265 +0000 UTC m=+3452.761629412"
Oct 30 21:59:02.275207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:02.275127    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qth9k_b941fd2b-fbbd-4343-81e1-9a0cdef17a10/global-pull-secret-syncer/0.log"
Oct 30 21:59:02.313232 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:02.313192    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-745kc_dccde73f-6cb5-467f-b650-48195b474db0/connection-checker/0.log"
Oct 30 21:59:02.342594 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:02.342558    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fdn45_e8e758fb-aeed-4b68-ba87-664edc7423ae/connection-checker/0.log"
Oct 30 21:59:02.372155 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:02.372062    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-m2xqt_14fc74c3-e4de-4506-a08b-f78dc63fe774/connection-checker/0.log"
Oct 30 21:59:02.563954 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:02.563823    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t48nt_4e6f376c-f439-45e5-9049-d7d097391b83/konnectivity-agent/0.log"
Oct 30 21:59:02.655861 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:02.655829    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-90.us-east-2.compute.internal_bb77e2dfdbc4a16f3bcede022ac475a7/haproxy/0.log"
Oct 30 21:59:07.284188 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:59:07.284058    4918 upgradeaware.go:427] Error proxying data from client to backend: readfrom tcp 127.0.0.1:49830->127.0.0.1:38631: write tcp 127.0.0.1:49830->127.0.0.1:38631: write: broken pipe
Oct 30 21:59:13.089068 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:13.086784    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-shzqw_a5946a39-1450-4d88-aae5-6f76b062fafb/cluster-monitoring-operator/0.log"
Oct 30 21:59:13.546319 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:13.546234    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvml7_4ef4d880-5d0a-4c18-9e26-e1819279529f/node-exporter/0.log"
Oct 30 21:59:13.581778 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:13.581740    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvml7_4ef4d880-5d0a-4c18-9e26-e1819279529f/kube-rbac-proxy/0.log"
Oct 30 21:59:13.608431 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:13.606368    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvml7_4ef4d880-5d0a-4c18-9e26-e1819279529f/init-textfile/0.log"
Oct 30 21:59:14.219855 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.219822    4918 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"]
Oct 30 21:59:14.323651 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.323595    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"]
Oct 30 21:59:14.323862 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.323736    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.443197 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.443086    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f3bed61-383a-40fd-b674-f433f97f4967-lib-modules\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.443197 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.443140    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vrq2\" (UniqueName: \"kubernetes.io/projected/1f3bed61-383a-40fd-b674-f433f97f4967-kube-api-access-2vrq2\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.443197 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.443196    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f3bed61-383a-40fd-b674-f433f97f4967-sys\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.443572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.443274    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f3bed61-383a-40fd-b674-f433f97f4967-proc\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.443572 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.443314    4918 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f3bed61-383a-40fd-b674-f433f97f4967-podres\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.545171 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.545126    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f3bed61-383a-40fd-b674-f433f97f4967-sys\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.545379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.545224    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f3bed61-383a-40fd-b674-f433f97f4967-proc\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.545379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.545266    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f3bed61-383a-40fd-b674-f433f97f4967-podres\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.545379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.545343    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f3bed61-383a-40fd-b674-f433f97f4967-lib-modules\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.545379 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.545375    4918 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vrq2\" (UniqueName: \"kubernetes.io/projected/1f3bed61-383a-40fd-b674-f433f97f4967-kube-api-access-2vrq2\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.545700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.545500    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f3bed61-383a-40fd-b674-f433f97f4967-proc\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.545700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.545567    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f3bed61-383a-40fd-b674-f433f97f4967-sys\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.545700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.545609    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f3bed61-383a-40fd-b674-f433f97f4967-podres\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.545700 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.545662    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f3bed61-383a-40fd-b674-f433f97f4967-lib-modules\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.555007 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.554975    4918 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vrq2\" (UniqueName: \"kubernetes.io/projected/1f3bed61-383a-40fd-b674-f433f97f4967-kube-api-access-2vrq2\") pod \"perf-node-gather-daemonset-wbg6j\" (UID: \"1f3bed61-383a-40fd-b674-f433f97f4967\") " pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:14.727018 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:14.726978    4918 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:15.090459 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:15.081342    4918 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"]
Oct 30 21:59:15.090459 ip-10-0-139-90 kubenswrapper[4918]: W1030 21:59:15.084850    4918 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1f3bed61_383a_40fd_b674_f433f97f4967.slice/crio-9d2376ad273a96c18b69257f1c126803c54c6ca4c1ec2d265131e88928b4abad WatchSource:0}: Error finding container 9d2376ad273a96c18b69257f1c126803c54c6ca4c1ec2d265131e88928b4abad: Status 404 returned error can't find the container with id 9d2376ad273a96c18b69257f1c126803c54c6ca4c1ec2d265131e88928b4abad
Oct 30 21:59:15.838825 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:15.838664    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-22hfw_148c1862-34bb-4b75-af99-96c64f2058b8/networking-console-plugin/0.log"
Oct 30 21:59:16.023064 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:16.021545    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vjht4/must-gather-jmmcm" event={"ID":"2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5","Type":"ContainerStarted","Data":"286ab974e2d4a2004f8c69928622126f96c37efe484028978f545d2e1a245628"}
Oct 30 21:59:16.034457 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:16.033255    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qm88q/must-gather-vgsm7" event={"ID":"19a277e1-4828-4800-a5b9-d514cd95bc94","Type":"ContainerStarted","Data":"dd5f5b46318629208d1551f5bda0f0acc12b5185f14444d211554bdd200f3a4c"}
Oct 30 21:59:16.042279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:16.042005    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j" event={"ID":"1f3bed61-383a-40fd-b674-f433f97f4967","Type":"ContainerStarted","Data":"2bb736eb1ad09d5ce5c32e96ad3bbf59f104f857b21700b6ae2a7d3991b9a9a4"}
Oct 30 21:59:16.042279 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:16.042111    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j" event={"ID":"1f3bed61-383a-40fd-b674-f433f97f4967","Type":"ContainerStarted","Data":"9d2376ad273a96c18b69257f1c126803c54c6ca4c1ec2d265131e88928b4abad"}
Oct 30 21:59:16.042662 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:16.042608    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:16.065335 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:16.063091    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j" podStartSLOduration=2.063075216 podStartE2EDuration="2.063075216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:16.061928287 +0000 UTC m=+3469.108487455" watchObservedRunningTime="2026-10-30 21:59:16.063075216 +0000 UTC m=+3469.109634364"
Oct 30 21:59:16.682245 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:16.682055    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/2.log"
Oct 30 21:59:16.687468 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:16.687372    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-22c7r_94385c98-9c6d-4fc1-9c48-38c9e5a95447/console-operator/3.log"
Oct 30 21:59:17.059668 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:17.059631    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vjht4/must-gather-jmmcm" event={"ID":"2fabd8a6-faae-4e27-a2df-12c8b3d8f2d5","Type":"ContainerStarted","Data":"f970bcf29cd942d8dd1790427864cddd98a7a4aab1ab3e905b8f4e6ad09de64e"}
Oct 30 21:59:17.063660 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:17.062924    4918 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qm88q/must-gather-vgsm7" event={"ID":"19a277e1-4828-4800-a5b9-d514cd95bc94","Type":"ContainerStarted","Data":"c095210a9af31acd10ebead1c3add054004ad6415ebeb5f02b55b33b96fe0361"}
Oct 30 21:59:17.088285 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:17.083134    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vjht4/must-gather-jmmcm" podStartSLOduration=2.470130666 podStartE2EDuration="20.083116464s" totalImagesPullingTime="17.612985798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:17.080749144 +0000 UTC m=+3470.127308289" watchObservedRunningTime="2026-10-30 21:59:17.083116464 +0000 UTC m=+3470.129675612"
Oct 30 21:59:17.102529 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:17.102470    4918 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qm88q/must-gather-vgsm7" podStartSLOduration=2.501664093 podStartE2EDuration="20.102453563s" totalImagesPullingTime="17.60078947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:17.100960622 +0000 UTC m=+3470.147519758" watchObservedRunningTime="2026-10-30 21:59:17.102453563 +0000 UTC m=+3470.149012705"
Oct 30 21:59:18.368553 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:18.368516    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-cwzsc_2c305cc1-5455-49d2-a20b-58b81ec38c3a/volume-data-source-validator/0.log"
Oct 30 21:59:20.504207 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:20.504177    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hsrbr_928df61d-6160-4586-acb9-d13cd90c1a72/dns/0.log"
Oct 30 21:59:20.527605 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:20.527570    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hsrbr_928df61d-6160-4586-acb9-d13cd90c1a72/kube-rbac-proxy/0.log"
Oct 30 21:59:20.651747 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:20.651715    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q7r7r_082e6a92-3eef-423d-859e-405adb01c3a1/dns-node-resolver/0.log"
Oct 30 21:59:21.582132 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:21.582096    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rzn4z_829cdf21-2065-4e39-902d-7b857a2d74bd/node-ca/0.log"
Oct 30 21:59:22.084414 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:22.083772    4918 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qzjln/perf-node-gather-daemonset-wbg6j"
Oct 30 21:59:29.169874 ip-10-0-139-90 kubenswrapper[4918]: E1030 21:59:29.169830    4918 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ltjmm" containerName="extractor"
Oct 30 21:59:29.922057 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:29.922026    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-fb99t_9a3c18ce-fc48-4be6-924c-5f4c89a8c8c1/discovery/0.log"
Oct 30 21:59:30.657934 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:30.657903    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5978974d9d-92d2b_7971b944-89b9-447c-96d8-8111a389680b/kube-auth-proxy/0.log"
Oct 30 21:59:30.798780 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:30.798748    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-d54n7_40508368-7d69-41e3-a2ee-0be190a20281/istio-proxy/0.log"
Oct 30 21:59:30.913051 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:30.912968    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-w9kjp_de5f2249-cc22-496c-b84a-8d1fcf7c5fc7/istio-proxy/0.log"
Oct 30 21:59:31.199919 ip-10-0-139-90 kubenswrapper[4918]: I1030 21:59:31.199843    4918 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d5fbcf987-vhd7f_7c27677e-bbe2-4457-b205-1faae8635cb0/router/0.log"